Need some clarification about beta/alpha testing on the developer console

Question

Background

The Android developer console has 3 tabs for publishing the app's apk file: alpha, beta and production, as shown here:

as I recall from one of Google IO lectures, one cool way to check how good is your app before making a 100% scale publishing, is to allow only a percentage of the users to download the app first. I think it's called "staged rollouts" , because you can rollout the publishing in case it had too many problems to be published to all.

My question

1. What is exactly the difference between them, especially between Alpha and Beta?

2. Only the production stage is available for people on the play store, right?

3. which one/s allow to publish only to specific people/percentage , and in which way do you do it?

4. which stage allows in-app billing, at least for testing ? I don't get why can't i test it out even before uploading the app.

5. in the percentage method, if I publish a new app version using the same way, will it first update for the people who were lucky enough to install the previous version?

Answer 1

See there are Four types of releases on play console for android developers;

1. Internal testing - It means you wanna share your app among the decided members. It means you have to put their Gmail into the testers list. This is usually for employees.

2. Alpha testing - Same as internal testing, but this time you can send your app to your friends, families, or on your smartphones.

3. Beta testing - This is a serious matter, this is public testing. But people cannot give reviews to your app. You can also limit how many users can install the app.

4. Production - This is the last release, now everybody can install your app, can give reviews, can report to google play if something bad is there. One more thing, now your app appears in play store search listing.

Now there is one more thing, open testing, closed testing, pausing tracks and stage rollouts, don't get confused by these terms.

1. Open testing - It means that you don't need to invite them, like no need to take their Gmail and add it to the user list. (Beta testing)

2. Closed testing - It means that you need to specifically add the user's Gmail into the list. (Alpha testing)

3. Staged rollout - This means we can provide the update to some percentage of users we currently have. Suppose 100 users are there and you rollout for 40% then only 40 users can update the app and after you confirm the app is working perfectly (using crash reports or reviews) then make it available for everyone.

4. Pause Track - See each testing has a track in play console. So if you click the pause track in beta testing then it will pause beta testing [changes occurs in few hours]. Same in internal and alpha. We usually pause test tracks when we release the app in production [usually I do].

Your app will not appear in search results until it is in production.

Answer 2

To answer your questions:

1.What is exactly the difference between them, especially between Alpha and Beta?

There isn't much difference between the two aside from the fact that you just start with a small number of testers for alpha testing and switch to a bigger group for beta

2.Only the production stage is available for people on the play store, right?

By default, only production is available on the Play Store. However, you can now add an option in your Play Store page for users to opt into an open Beta program. Link

3.which one/s allow to publish only to specific people/percentage , and in which way do you do it?

You can do that for both. To do alpha,beta testing you need to send invites to people on their google+ accounts so that they can access your app and be able to download it. The invites is usually in the form of a link directing them to your app on the play store which is only visible to them after they accept the invite

4.which stage allows in-app billing, at least for testing ? I don't get why can't i test it out even before uploading the app.

You can do in-app billing for both alpha,beta testing. Check the link: http://developer.android.com/google/play/billing/billing_testing.html

5.in the percentage method, if I publish a new app version using the same way, will it first update for the people who were lucky enough to install the previous version?

As far as my experience goes, i have seen people who were testers get the update for the app first than everyone else. But i am not so sure about how does it work exactly.

Hope this helps.

Answer 3

Difference between Alpha and Beta testing (Software/Apps Testing)

Alpha Testing is conducted within the organization and tested by an individual developer or a team of developers or testers. This testing is closed for public.

Beta testing is conducted by the end users who are not programmers, software engineers or testers. This testing may be open for public.

Open or Closed Testing

Closed Alpha Testing: This testing is conducted within the organization and restricted by email or group invitations only. Those who are added onto the list of alpha testers on Google Play Store have access to test app.

Open Alpha Testing: Those who have Opt-in link of app have access to test the app. This testing is not restricted by email or group invitations. You can limit the number of testers on Google Play Store.

Closed Beta Testing: This testing is conducted outside the organization and restricted by email or group invitations. Those who are added onto the list of beta testers on Google Play Store have access to test app.

Open Beta Testing: This testing is conducted outside the organization and it is not restricted by email or group invitations. This testing is conducted in real time as the app will be made open for public to install. You can limit the number of testers on Google Play Store.

Generally, alpha testing is first conducted by developers or testers within the organization. Beta testing comes after and is conducted by non-technical or end users outside the organisation. Finally, after completing the testing, the app goes to production.

In app purchase

Licensed users may purchase your in-app products without any resulting charges to the user. Test purchases can be only used in both alpha and beta releases.

To add user license: Google play Console -> Settings -> Developer Account -> Account details -> License Testing

Learn more about testing

Answer 4

@user2511882's answer is great, but there are more information to add.
First of all, alpha and beta channels are designed to test multiple version of app. You can test only one version (identified by versionCode) on each channel, so alpha and beta channels provide two test tracks.

However they have a little difference on test type. You can run closed alpha and open beta test simultaneously, but it's not possible to run two open alpha/beta test nor open alpha and closed beta test.

Another difference is that alpha test APK's version code should be higher than beta test one. If you upload higher version of APK on beta test channel, alpha test will automatically be closed. (This principle applies for production APK too. APK version code should be 'Production < Beta < Alpha'.)

For more details, check this help.