End-to-end SDK generation and publishing with Bitbucket Pipelines
This tutorial includes the following SDK languages and versions:
TypeScript Java Python C# Go PHP ✅ ✅ ✅ ✅ ✅ ✅
This tutorial will guide you through the process of setting up an end-to-end automated workflow to generate, manage, and publish SDKs using liblab and Bitbucket Pipelines. You'll learn how to configure a control repo to store your API specifications, define SDK repos for different programming languages, and set up tokens to automate the deployment of your SDKs to package managers such as npm, PyPI, and Maven.
By the end of this tutorial, you'll have an automated pipeline that ensures your latest API updates are made available quickly to your users through automatically generated and published SDKs.
Prerequisites
To follow this tutorial, you'll need the following:
- liblab CLI installed and logged in.
- A Bitbucket account.
- An API specification in OpenAPI or Swagger format.
- Package manager accounts with appropriate publishing permissions for npm, PyPI, Maven, etc.
Steps
This tutorial covers the following steps:
- Create a control repo
- Create SDK repos
- Create liblab and Bitbucket tokens
- Create package manager tokens
- Update your spec and config file
- Merge SDK PRs
- Create a release in Bitbucket
- Confirm SDK publication
1. Create a Control Repo
To begin, set up a control repo as the main storage for your liblab config file, any hooks code, and the API spec.
Storing the OpenAPI spec directly in the control repo isn't required. You can link to an external repo through the liblab.config.json
file.
To streamline setup, use the control-repo-template. For more details, access the control repo documentation. The control-repo-template provided by liblab includes:
File | Description |
---|---|
bitbucket-pipelines.yml | Automates SDK generation and pull request (PR) creation when changes occur in specified files and directories, such as liblab.config.json , spec/ , hooks , and customPlanModifiers . It triggers on pushes to the main branch and on pull requests. This pipeline uses a Node.js environment to detect changes, install liblab, and create a PR in Bitbucket if updates are found. Configure LIBLAB_TOKEN and LIBLAB_BITBUCKET_TOKEN to ensure correct execution. |
liblab.config.json | Sample liblab config file containing the settings used by the liblab CLI to generate SDKs. |
spec.json | Example OpenAPI spec based on Petstore. The liblab.config.json file is refering this OpenAPI spec. |
.github/workflows | Directory with two GitHub Actions. One creates PRs for your SDKs when your API spec or config file changes, and another polls for liblab updates, creating PRs for SDK updates. These GitHub Actions are adapted for use with Bitbucket. |
Since the control-repo-template is hosted on GitHub, follow these steps to integrate it with Bitbucket:
-
Log in to your Bitbucket account.
-
Create or join a Bitbucket workspace.
-
In the navigation bar, select Create > Repository:
- Choose an existing project or create a new one.
- Enter a Repository name.
- Uncheck Private repository.
- Select No for Include a README and Include .gitignore.
- Set
main
as the Default branch name. - Click Create repository.
-
Clone the GitHub-hosted control-repo-template to your local machine:
git clone https://github.com/liblaber/control-repo-template
- Navigate to the cloned repo on your machine, then add your new Bitbucket repo as a remote:
cd control-repo-template
git remote rename origin old-origin
git remote add origin https://<your-bitbucket-organization>/<your-bitbucket-project>/control-repo.git
- Push the control-repo-template to your Bitbucket repo:
git push -u origin main
If it's your first time using Bitbucket, a new windows will show up asking you to connect with Bitbucket.
This is what you should have on your Bitbucket project:
2. Create SDK repos
Each SDK you create requires a dedicated repo for storage and publication. These SDK repos should be created before you can publish your SDKs. You can set up these SDK repos manually or use one of the liblab templates.
To get started, create a new Bitbucket project for each SDK language, or use liblab's available templates:
If you're using a liblab template, follow the steps in Step 1 to push the template content to your Bitbucket repos.
Each liblab template includes a Bitbucket pipeline file, automating the SDK's build and deployment process to the package manager service.
For more information on SDK repos, access the SDK repos documentation.
After setting up the SDK repos, you should have the following:
3. Create liblab and Bitbucket tokens
The Bitbucket pipeline in the control repo, which generates SDKs, requires two tokens:
LIBLAB_TOKEN
: A liblab token that authenticates the liblab CLI.LIBLAB_BITBUCKET_TOKEN
: A Bitbucket access token that allows the liblab CLI to push content to your SDK repos.
3.1 Create a liblab token
To create the liblab token, follow these steps:
-
On your machine, use the liblab CLI to generate a liblab token:
Terminalliblab token create LIBLAB_TOKEN
You can replace
LIBLAB_TOKEN
with any preferred name. If you're generating SDKs for multiple APIs, you can either use the same token for all APIs or generate separate tokens for each.TerminalToken successfully generated, it will be valid for 90 days
-TOKEN-----------------------------------------
liblab_HYsJnYb5Ch6EhlSdlNuF_qurwUEGQg75fw9H8bEE
-----------------------------------------------noteBy default, the token expires after 90 days. You can extend this to up to 364 days by using the
--durationInDays
parameter.See the CLI documentation for more information about the
liblab token
command. -
Copy the generated token string (
liblab_...
) from the central line. This token will allow liblab to authenticate during CI/CD when the pipeline runs. -
Add the token to your Bitbucket control-repo project. Access your Bitbucket project, navigate to Settings > CI/CD > Variables and click Add variable. Set Key to
LIBLAB_TOKEN
and paste the token in the Value field, then click Add variable.
3.2 Create a Bitbucket access token
To enable the Bitbucket pipeline to run the liblab CLI and push changes to your SDK repos, you need a Bitbucket access token. Follow these steps to create a project access token:
liblab recommends using a Project access token. If the control repo and SDK repos are in the same project, this limits the token's access scope to necessary repos.
For non-premium accounts, consider creating a repository access token for the SDK repo. However, this restricts usage to one SDK repo.
- In Bitbucket, go to Projects and select the project containing the control repo and SDK repos.
- In the left sidebar, choose Project settings.
- Select Access tokens.
- Set Label to
LIBLAB_BITBUCKET_TOKEN
. - Assign the following permissions:
- Repositories:
- Read
- Write
- Pull Request:
- Read
- Write
- Repositories:
- Click Create and copy the generated token. This token will authenticate and push SDK code to your SDK repos from the control repo.
- Add the token as a repository variable in your control repo, with the name
LIBLAB_BITBUCKET_TOKEN
.
This is what you should have set up so far:
4. Create package manager tokens
Depending on what SDK language you are using, you will need to create additional secrets.
- TypeScript
- Python
- C#
- Java
- Go
- PHP
To publish to npm, you will need an npm access token. You can generate a new access token from your user settings on npm.
-
Select the Access Tokens tab, drop down the Generate New Token button, then select Granular Access Token.
-
Fill in all the required details for the token such as the name and expiry.
-
Make sure that this token has read and write permission to publish packages. If the package already exists, you can scope this token to just that package, otherwise this token needs read and write for all packages.
-
Once the token has been created, make a copy of it.
-
In your TypeScript SDK repo, add this token as a repository variable named
NPM_TOKEN
.
You can learn more about creating npm tokens in the npm access tokens documentation.
To publish to PyPI you will need an API token. You can generate a new API token from your account settings page on PyPI.
-
Select Account settings, scroll to the API tokens section, then select the Add API token button.
-
Give the token a name.
-
Select the scope for the token. If the package already exists, select that package for the scope, otherwise select Entire account (all packages).
-
Select the Create token button.
-
Once the token has been created, make a copy of it.
-
In your Python SDK repo, add this token as a repository variable named
PYPI_TOKEN
.
You can learn more about creating PyPI tokens in the PyPI API tokens documentation.
To publish to NuGet, you will need a NuGet API key. You can generate an API key from your account on NuGet.
-
Select your name on the top right, then select the API Keys.
-
From the API Keys page, expand the Create section.
-
Give your API Key a name, and set the required expiry and package owner.
-
Set the scope for this API Key.
- If the package already exists, select Push and Push only new package versions. Then in the Select Packages section, select the package you want to publish updates for.
- If the package does not exist, select Push and Push new packages and package versions, then in the Glob pattern, put
*
. This will allow you to push a new package.
-
Select the Create button.
-
Once the API Key has been created, expand the Manage section, and make a copy of it using the Copy button.
-
In your C# SDK repo, add this token a repository variable named
NUGET_TOKEN
.
You can learn more about creating NuGet API Keys in the NuGet API Keys documentation.
To publish to the Maven Central Repository, you will need Maven credentials and a GPG key with a passphrase to sign your artifacts.
As of March 12th, 2024, the Central Portal became the default publishing server for Maven packages. At the moment, we do not support automatic publishing through the Legacy OSSRH.
Central Portal Setup
The Central Portal Account Documentation explains how to create an account. An account is required for claiming namespaces, generating credentials and managing package deployments.
Namespace Registration
Your namespace is the crucial prerequisite for publishing a package to the Central Repository as it is the groupId
of the package. The Central Repository Namespace Documentation provides detailed instructions for claiming a namespace.
The namespace verification process is not instant.
- For own domain namespaces (e.g.
com.liblab
), an additional step is required, which involves setting up the verification key as a DNS record in order for the domain name to be verified by the Central Repository.
Once your namespace is verified, the next step is to generate your Maven credentials.
Generate Maven Credentials
-
Head to the Central Portal at central.sonatype.com, and either create an account, or sign in.
-
Select your email on the top right, then select View Account.
-
From the Account page, select the Generate User Token button.
-
When the dialog appears, confirm that you want to generate a new token, and credentials will be generated.
-
In your Java SDK repo, add value of the
Username
field as a new repository variable calledMAVEN_USERNAME
and the value of thePassword
field as a repository variable calledMAVEN_PASSWORD
.
You can learn more about creating Central Portal credentials in the Central Portal Documentation.
Generate GPG Key
The Central Repository requires all artifacts to be signed with PGP, such as using a GPG key. You will need to generate your own key pair, distribute it to the key server and obtain the private key, then set this as an other repository variable.
The following steps will guide you through the process of creating a GPG key using GnuPG. If you already have a GPG key, you can skip this step. If you want to use a different tool to generate your GPG key, you will need to follow the relevant documentation for that tool.
-
Install GnuPG if you don't have it installed.
-
Initialize the key generation prompt by running:
Terminalgpg --gen-key
-
Follow the instructions from
gpg
, and choose a passphrase -
Once the key is generated, you will need to get the key id. This is the 40-character hex string on the second line of the
pub
section in the output from generating the key:Terminalpublic and secret key created and signed.
pub ed25519 2024-05-07 [SC] [expires: 2027-05-07]
AB1C2D3456EF78A90BC12D34567890123456789E
uid Exciting Soda <[email protected]>
sub cv25519 2024-05-07 [E] [expires: 2027-05-07] -
Send the key to your key server, such as
openpgp
by running the following command:Terminalgpg --keyserver keys.openpgp.org --send-keys <key-id>
Replace
<key-id>
with the key id you obtained in the previous step. -
Get the private key using the following command:
Terminalgpg --export-secret-keys --armour <key-id>
Replace
<key-id>
with your key id.Terminal➜ ~ gpg --export-secret-keys --armour AB1C2D3456EF78A90BC12D34567890123456789E
-----BEGIN PGP PRIVATE KEY BLOCK-----
mDMEXEcE6RYJKwYBBAHaRw8BAQdArjWwk3FAqyiFbFBKT4TzXcVBqPTB3gmzlC/U
b7O1u120JkFsaWNlIExvdmVsYWNlIDxhbGljZUBvcGVucGdwLmV4YW1wbGU+iJAE
ExYIADgCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AWIQTrhbtfozp14V6UTmPy
MVUMT0fjjgUCXaWfOgAKCRDyMVUMT0fjjukrAPoDnHBSogOmsHOsd9qGsiZpgRnO
dypvbm+QtXZqth9rvwD9HcDC0tC+PHAsO7OTh1S1TC9RiJsvawAfCPaQZoed8gK4
OARcRwTpEgorBgEEAZdVAQUBAQdAQv8GIa2rSTzgqbXCpDDYMiKRVitCsy203x3s
E9+eviIDAQgHiHgEGBYIACAWIQTrhbtfozp14V6UTmPyMVUMT0fjjgUCXEcE6QIb
DAAKCRDyMVUMT0fjjlnQAQDFHUs6TIcxrNTtEZFjUFm1M0PJ1Dng/cDW4xN80fsn
0QEA22Kr7VkCjeAEC08VSTeV+QFsmz55/lntWkwYWhmvOgE=
=iIGO
-----END PGP PRIVATE KEY BLOCK----- -
Copy the output from this command, and add it to a new repository variable called
GPG_PRIVATE_KEY
. -
Create another repository variable called
GPG_PASSPHRASE
, and set this to the passphrase you chose when generating the key.
Go package management differs from many other languages in that it does not require API tokens for publishing to the Go Packages repository. Unlike some ecosystems where API tokens are necessary to interact with package registries, Go relies on its module proxy system, which integrates seamlessly with GitHub.
As explained in the official publishing tutorial on Go's website, packages are published to the Go Packages repository by tagging a release on GitHub. The Create a release in GitHub section later in this tutorial provides a step-by-step guide on how to create a release in GitHub which will automatically trigger the publishing of the package to the Go Packages repository. Upon creating a release, it should not take longer than one hour for the package to become available on the Go Packages repository.
To publish your PHP SDK to Packagist, you will first need an account on Packagist and your Packagist API Token.
-
Create an Account on Packagist or login to Packagist using your GitHub account.
-
Obtain your Packagist API Token from your profile settings
-
In your PHP SDK repo, add this token as an repository variable named
PACKAGIST_TOKEN
and set the repository variablePACKAGIST_USERNAME
to your packagist username as it appears on your profile. -
Submit your package
- Log in to your Packagist account.
- Go to the Submit tab.
- Enter the URL of your SDK's repository.
- Select Check to validate the repository and Submit to add your package to Packagist.
You should now have the following complete setup:
5. Update your spec and config file
After setting up your SDK repos, the next step is to update your control repo with your API spec and configure the liblab.config.json
file.
Follow these steps to update your API spec and trigger the release of new SDKs to their respective repos:
-
Open the control repo you cloned in Step 1 on your machine.
-
Replace the control repo's sample OpenAPI spec (
spec.json
) with your API spec. To follow this tutorial, you can keep the existing sample file.- If you have a local OpenAPI spec, replace the
spec.json
file directly. - For a remotely hosted API spec:
- Update the
specFilePath
field in theliblab.config.json
file with the URL of your remote API spec. - Delete the
spec.json
file from the repo if you're using a remote spec.
- Update the
- If you have a local OpenAPI spec, replace the
-
Open the
liblab.config.json
file in your control repo and update the following fields:sdkName
: Enter the name of your SDK.apiName
: Enter the name of your API.apiVersion
: Specify the API version.languages
: List the SDK languages to generate, such as TypeScript, Python, and Go.
-
In
liblab.config.json
, locate thelanguageOptions
section to configure each SDK language:-
Remove configurations for any unused languages.
-
For each language, update the
githubRepoName
field with the name of the corresponding Bitbucket SDK repo. liblab will use it to create the pull request with the new SDK code. -
Set the
sdkVersion
for each SDK. -
Configure the package manager options as follows:
- TypeScript
- Python
- C#
- Java
- Go
- PHP
- Set the
pypiPackageName
option to the name of your PyPI package. This needs to be unique.
- Set the
packageId
option to the name of your NuGet package.
- Set the
groupId
option to your namespace verified by the Maven Central Portal. - Set the
homepage
option to the valid public URL of your SDK homepage. - Add at least one developer to the
developers
option. - (Recommended) Set the
artifactId
option. If not set, it will default to the kebab-cased version ofsdkName
.
- Set the
goModuleName
option to the name of your Go module. This should match the SDK repo name, for example,bitbucket.org/myorg/go-sdk
. - Ensure your license is set.
note
The Go Packages License Policy requires that all packages published to the Go Packages repository must include a recognized license file. It is important to note that if the package does not include a recognized license file, only limited package information will be displayed on the Go Packages repository.
For more information on how to specify a license for your SDK, see the license config file option.
- Set the
packageName
option to the name of your PHP package. It must be in the formatvendor/packageName
.
-
In the
publishing
section, setgithubOrg
to the Bitbucket workspace that contains the SDK repos.
tipAdditional optional settings are available in the liblab config file documentation.
Your
liblab.config.json
file should look like this:liblab.config.json{
"sdkName": "test-sdk",
"apiVersion": "1.0.0",
"apiName": "test-api",
"specFilePath": "spec.json",
"languages": [
"csharp",
"python",
"typescript",
"go",
"php",
"java"
],
"languageOptions": {
"csharp": {
"liblabVersion": "2",
"packageId": "Test.SDK",
"githubRepoName": "csharp-sdk",
"sdkVersion": "1.0.0"
},
"python": {
"liblabVersion": "2",
"pypiPackageName": "test-sdk",
"githubRepoName": "python-sdk",
"sdkVersion": "1.0.0"
},
"typescript": {
"npmName": "test-sdk",
"npmOrg": "myorg",
"githubRepoName": "typescript-sdk",
"sdkVersion": "1.0.0"
},
"java": {
"groupId": "com.myorg",
"artifactId": "test-sdk",
"homepage": "https://myorg.com",
"githubRepoName": "java-sdk",
"sdkVersion": "1.0.0",
"developers": [
{
"name": "John Doe",
"email": "[email protected]",
"organization": "My Organization",
"organizationUrl": "https://myorg.com"
}
]
},
"go": {
"goModuleName": "bitbucket.org/myorg/go-sdk",
"githubRepoName": "go-sdk",
"sdkVersion": "1.0.0",
},
"php": {
"packageName": "myorg/test-sdk",
"githubRepoName": "php-sdk",
"sdkVersion": "1.0.0"
},
},
"publishing": {
"githubOrg": "myorg"
}
} -
-
Commit the changes you performed to the
liblab.config.json
and OpenAPI spec files. -
Push the updates to your control repo to trigger the pipeline, generating your SDKs and creating pull requests in the SDK repos.
6. Merge SDK PRs
In the control repo, navigate to Pipelines to check the pipeline and job status. Once the pipeline in the control repo completes, a pull request PR will be automatically generated in each SDK repo.
After the job completes successfully in the control repo, review the PRs generated by liblab in each SDK repo. To view these PRs, select Pull requests in each SDK repo.
If you don't see the PRs, check the control repo's Pipelines for any errors during the SDK generation process. You can also refer to the common errors section for troubleshooting.
To complete the process:
- Open each PR in the SDK repos and review the changes to ensure the SDK was generated correctly.
- If all changes are correct, approve the PRs and merge them into the main branch.
7. Create a release in Bitbucket
Bitbucket does not provide a native way of automatically generating release notes. You can use a third-party solution or create release notes using Jira. Access the Atlassian tutorial page for more information.
8. Confirm SDK publication
After completing the publishing process, check the package manager services related to the SDKs you have created, such as npm, PyPI, and Maven, to verify that your SDK has been successfully published.
Your setup is now fully automated to generate and publish SDKs with liblab and Bitbucket pipelines, ensuring your latest API updates are quickly available to your users.
If you don't see the SDK in the package manager, check the Pipelines section in each SDK repo to troubleshoot any errors during the publishing process.
Common errors
If your repos are not configured correctly, you may see one of the following errors:
-
If the generate SDKs Action fails with this error:
Bitbucket pipeline outputError: Secrets LIBLAB_TOKEN and LIBLAB_BITBUCKET_TOKEN are required
Then you haven't set up the
LIBLAB_TOKEN
orLIBLAB_BITBUCKET_TOKEN
secrets correctly. Check that the tokens are set correctly, and that the tokens are valid. -
If the pipeline generates the SDKs but fails to create a pull request in the SDK repo, presenting an error similar to the next one:
Bitbucket pipeline outputError occurred while creating PR for Python: Access to push changes to branch v1.0.0-1730738768515 is forbidden
Then, your
LIBLAB_BITBUCKET_TOKEN
access token does not have the necessary permissions. Remember, you have to create a workspace or project access token. Only use the Repository access token if you are using the free account. Don't use the Bitbucket app password in the access token place.