Skip to main content

End-to-end SDK generation and publishing with Jenkins Pipelines

This tutorial includes the following SDK languages:

TypeScriptJavaPythonC#GoPHP
✅✅✅✅✅✅

This tutorial guides you through an end-to-end process for generating, managing, and publishing SDKs using Jenkins Pipelines and liblab. By following this tutorial, you’ll learn how to automate the CI/CD workflow for your SDKs, from repository setup to SDK publication in package managers like npm, PyPI, and Maven.

Prerequisites​

To follow this tutorial, you’ll need the following:

Steps​

This tutorial covers the following steps:

  1. Create a control repo
  2. Create SDK repos
  3. Create liblab and GitHub tokens
  4. Create package manager tokens
  5. Configure Jenkins
  6. Update your spec and config file
  7. Check the Jenkins pipeline
  8. Merge SDK PRs
  9. Create a release in GitHub
  10. Confirm SDK publication

1. Create a control repo​

The first step is to create a control repo, which is used as the central repo to store your liblab config file, any hooks code, and the API spec.

API spec

You don't need to store the OpenAPI spec in the repo. You can link your OpenAPI spec from other repos directly to the liblab.config.json file.

Version Control Systems

When using Jenkins to control your CI/CD, you can choose to use GitHub, GitLab, or BitBucket.

This tutorial uses GitHub to host the necessary repos.

For this tutorial, you can use the control-repo-template created by liblab. Access the control repo documentation for more information. To use this repo, follow the steps:

  1. Create a new repo using the control-repo-template.
  2. Define the new repo name and add a description. The repo can be Public or Private.
  3. Click Create repository.

The new repo will contain the following files:

FileDescription
JenkinsfileA pipeline configuration file for automating SDK generation and pull request creation when specific files or directories, such as liblab.config.json, spec/, hooks/, or customPlanModifiers/, are modified. It defines steps to validate environment variables, install Node.js and dependencies, detect file changes, build SDKs using liblab, and create pull requests.
liblab.config.jsonAn example of liblab configuration file. It contains the configuration used by the liblab CLI to generate SDKs.
spec.jsonAn OpenAPI spec example based on the Petstore. The liblab.config.json file is refering this OpenAPI spec.

The other existing files are not used in this tutorial.

This is what you should have on your repo project:

2. Create SDK repos​

Each SDK you create needs its own repo for storage and publication. These SDK repos should be created before you can publish your SDKs. You can create these repos manually or use one of the liblab templates.

Create new GitHub projects for each SDK language or use liblab's templates:

Each liblab template includes a GitHub Action that automates the SDK build and deployment process to the package manager service.

note

You can learn more about the SDK repos by accessing the SDK repos documentation.

After creating the repos, this is what you should have:

3. Create liblab and GitHub tokens​

To manage your CI/CD process with Jenkins, you must configure the following secrets to enable interactions with GitHub and access liblab features:

  • liblab Token: Authenticates the liblab CLI to execute commands and manage SDK generation.
  • GitHub Token: Grants Jenkins and the liblab CLI permission to push updates to your SDK repos.

3.1 Create a liblab token​

To create the liblab token, follow the steps below:

  1. Using the liblab CLI on your machine, run the following command to generate a liblab token:

    Terminal
    liblab token create LIBLAB_TOKEN

    You can replace LIBLAB_TOKEN with any name of your choice. If you're generating SDKs for multiple APIs, you can either use the same token for all of them or create separate tokens for each API.

    Terminal
    Token successfully generated, it will be valid for 90 days

    -TOKEN-----------------------------------------
    liblab_td7MY4WfsE6Df4BRfyhoma7uONeOrSWmvxtSRPDj
    -----------------------------------------------
    note

    By default, the token expires after 90 days. You can extend this to 364 days by passing the --durationInDays parameter.

    Learn more about the liblab token command in the CLI documentation.

  2. Copy the generated token. You should copy only the string liblab_... at the central line. The token will allow liblab to authenticate during the CI/CD pipeline run.

Securely store the token. You'll need it later to configure credentials in Jenkins.

3.2 Create a GitHub token​

To enable the Jenkins pipeline to identify changes in your control repo, run the liblab CLI, and push changes to your SDKs repos, you need to generate a GitHub Fine Grained Access Token:

  1. Navigate to your GitHub profile and click Settings.

  2. In the sidebar, click Developer Settings.

  3. At the Developer Settings page, select Personal access tokens > Fine-grained tokens.

  4. Click Generate new token.

  5. Name the token GITHUB_TOKEN and set its expiration.

    Expiration

    It's helpful to create a calendar reminder to refresh the token before it expires or use the No expiration option.

  6. Under ** Repository Access **, select ** Only select repositories **. Choose the repos created in Step 1 and Step 2.

  7. Set the following repo permissions:

    • Commit Statuses: Read/Write.
    • Contents: Read/Write.
    • Metadata: Read.
    • Pull Requests: Read/Write.
    • Workflows: Read/Write.

Repository permissions

  1. Click Generate token.
CI/CD documentation

Access the Pull request creation documentation page for further information and detailed guidance on creating the token.

Once created, copy the GITHUB_TOKEN and save it in a secure location for use in the upcoming steps. For more details on how the GITHUB_TOKEN will be used to create PRs, access the liblab documentation.

4. Create package manager tokens​

Depending on your SDK language, you will need to create additional secrets.

To publish to npm, you will need an npm access token. You can generate a new access token from your user settings on npm.

  1. Select the Access Tokens tab, drop down the Generate New Token button, then select Granular Access Token.

    The access tokens page in npm settings

  2. Fill in all the required details for the token such as the name and expiry.

  3. Make sure that this token has read and write permission to publish packages. If the package already exists, you can scope this token to just that package, otherwise this token needs read and write for all packages.

  4. Once the token has been created, make a copy of it.

  5. In your TypeScript SDK repo, add this token as an Actions secret named NPM_TOKEN.

    The token set as an actions secret

note

You can learn more about creating npm tokens in the npm access tokens documentation.

You should now have the following complete setup:

5. Configure Jenkins​

You can install Jenkins using different approaches and host it locally or in the cloud. In this tutorial, Jenkins runs in a Docker container, and a Docker composer file defines the configurations. The following steps describe how to run and configure Jenkins.

5.1 Run Jenkins using Docker composer​

For this tutorial, Jenkins will run in a Docker and a Docker composer is used to define the configurations. The following code snippet shows the necessary code to use in the docker-compose.yml file.

docker-compose.yml
version: '3.8'

services:
  jenkins:
    image: jenkins/jenkins:lts
    container_name: jenkins
    ports:
      - '8080:8080'
      - '50000:50000'
    volumes:
      - ${HOME}/jenkins:/var/jenkins_home
    privileged: true
    user: root
Volume Configuration

Replace ${HOME} with an existing directory path on your machine to ensure Jenkins data is persisted locally.

In the code above, volume links the host machine's ${HOME}/jenkins directory to the container's /var/jenkins_home. This ensures Jenkins data, such as jobs and configurations, remains intact even if the container is stopped or removed. The privileged: true setting grants the container additional permissions. The user: root setting allows the container to run as the root user, enabling more control for system-level operations.

To start the Jenkins service, follow these steps:

  1. In a new directory, separate from those used in Step 1 and Step 2, create a file named docker-compose.yml.

  2. Copy and paste the provided code into the file and save it.

  3. Open a terminal in the directory containing the docker-compose.yml file and run the following command:

    docker-compose up -d

    Your terminal should display output similar to this:

    $ docker-compose up -d
    [+] Running 1/1
    ✔ Container jenkins        Started

  4. Once the Jenkins service is running, it will be accessible in your browser at http://localhost:8080.

  5. To log in, you need the Jenkins Admin password. Follow these steps to retrieve it:

    1. Access the terminal of the Docker container running Jenkins:

      docker exec -it jenkins bash

    2. Inside the Docker terminal, run the following command to retrieve the Admin password:

      cat /var/jenkins_home/secrets/initialAdminPassword

    The password will be displayed in the terminal. Save it securely, as you'll need it to log in to Jenkins.

5.2 Run Jenkins and install the plugins​

To get started with Jenkins, follow the steps:

  1. Access Jenkins through your browser using the address http://localhost:8080.

  2. Use the admin password from Step 5.1 to unlock Jenkins. For future logins, you'll also need to use the admin user with the admin password.

    Jenkins unlok screen

    Additional Users

    Creating additional users is not necessary for this tutorial. You can perform all actions using the Admin user.

  3. Select the Install suggested plugins option. These are the essential plugins to ensure Jenkins will run correctly.

    Select the suggested plugins installation

  4. Jenkins will download and install the selected plugins. Once the installation is complete, you'll be redirected to the Jenkins dashboard.

    Jenkins dashboard screen

To set up your Jenkins environment, you need to install three additional plugins. Follow these steps to complete the installations:

  1. From the Jenkins dashboard, navigate to Manage Jenkins -> Plugins -> Available plugins.
  2. Search for and select the following plugins:
    • Docker plugin
    • Docker Pipeline
    • Basic Branch Build Strategies Plugin
  3. Once the plugins are selected, click Install.
  4. After the installation is complete, Jenkins will restart automatically.
note

During the restart process, the Jenkins container may stop temporarily. If you cannot access the dashboard at http://localhost:8080 and see a Page Not Found error, restart the Jenkins container by running the following command, as described in Step 5.1:

docker-compose up -d

5.3 Add your credentials​

After installing all the necessary plugins, the next step is to add the LIBLAB_TOKEN and GITHUB_TOKEN as credentials in Jenkins. These credentials are used by Jenkins to interact with your GitHub repos (both the control repo and SDK repos) and to use the liblab CLI. Follow the steps below to configure them:

  1. From the Jenkins dashboard, go to Manage Jenkins -> Credentials.
  2. Under Stores scoped to Jenkins, select System.
  3. Under the Domain, choose Global credentials.
  4. Click + Add Credentials.
  5. Fill in the form for each of the credentials as described below:
  • Kind: Select the Secret text option.
  • Secret: Paste the LIBLAB_TOKEN from Step 3.
  • ID: Use ID equal LIBLAB_TOKEN.
  • Description: Add a meaningful description to help identify this credential later.

At the end, your credentials should look similar to the following image.

Jenkins credentials dashboard

5.4 Create a Jenkins pipeline​

Now that your environment is configured, you can create a pipeline to manage your SDK projects' CI/CD process. Follow these steps to create the pipeline:

  1. From the Jenkins dashboard, click New Item.

  2. Enter a name for the pipeline, select Multibranch Pipeline, and click OK.

  3. On the pipeline configuration page, perform the following steps:

    1. In the Display Name field, add a name for the pipeline.
    2. In the Description field, provide a meaningful description of the pipeline’s purpose.
    3. For Branch Sources, select GitHub as the source. A new configuration form will appear.
      1. For Credentials, choose the username and password (github-credentials) you added in Step 5.3.
      2. Paste the control repo URL into the Repository HTTPS URL field.
      3. (Optional) Configure branch strategies if needed. For this tutorial, you can leave the default settings.
    4. For Build Configuration, select by Jenkins for the Mode and add Jenkins for the Script Path field.
    5. For the Scan Repository Triggers, set Interval to 10 minutes to define how often Jenkins should scan the repo for changes.
    Webhooks and GitHub plugins

    Instead of setting a scan interval, you can configure GitHub Webhooks for automatic triggering or use GitHub plugins to enhance Jenkins-GitHub integration.

  4. Click Save.

Once saved, Jenkins will scan the repo for existing branches and pull requests. It will automatically trigger a build if it finds a Jenkinsfile.

Jenkins scanning the repo

Since you’re using the control repo provided by liblab, Jenkins will confirm the presence of a Jenkinsfile and start the pipeline job. However, the pipeline will fail initially because the liblab.config.json file has not yet been updated with the SDK repos and organizations.

Jenkins pipeline console with error while building the SDKs and creating the pull request

6. Update your spec and config file​

After Configuring Jenkins, the next step is to update your control repo with your API spec and configure the liblab.config.json file.

Follow the steps below to update your API spec and trigger the release of new SDKs to the corresponding repos:

  1. Open the control repo in your preferred code editor.

  2. Replace the control repo API spec. The control repo includes a sample API spec (spec.json), which you can use to continue following this tutorial. However, you can replace the existing spec.json with your API spec.

    • If you have a local API spec, replace the spec.json file with your own.
    • If your API spec is hosted remotely:
      1. Update the specFilePath field in the liblab.config.json file with the URL of your remote API spec.
      2. Delete the existing spec.json file from the repo if you are using a remote spec.

  3. Open the liblab.config.json file and update the following fields:

    • sdkName: Set the name of your SDK.
    • apiName: Set the name of your API.
    • apiVersion: Specify the API version you're working with.
    • languages: List the SDK languages you want to generate, such as TypeScript, Python, Go, etc.

  4. In the liblab.config.json, locate the languageOptions section and update the options for each SDK language you are generating:

    1. Remove any unused languages. If there are languages you don't need, remove their configuration.

    2. For each language, update the githubRepoName. liblab will use it to identify the GitHub project that matches the name of the corresponding SDK repo.

    3. Update the sdkVersion field to specify the version number for each SDK.

    4. Configure the specific fields for each package manager:

      • Set the npmName option to the name of your npm package.
      • Set the npmOrg option to the name of your npm organization.

    5. In the publishing section, set the githubOrg as the GitHub organization from the corresponding SDK repos.

      tip

      You can configure additional optional settings. Access the liblab config file documentation to learn more.

    After finishing, your liblab.config.json file should look like this:

    liblab.config.json
    {
      "sdkName": "test-sdk",
      "apiVersion": "1.0.0",
      "apiName": "test-api",
      "specFilePath": "spec.json",
      "languages": [
        "csharp",
        "python",
        "typescript",
        "go",
        "php",
        "java"
      ],
      "languageOptions": {
        "csharp": {
          "liblabVersion": "2",
          "packageId": "Test.SDK",
          "githubRepoName": "csharp-sdk",
          "sdkVersion": "1.0.0"
        },
        "python": {
          "liblabVersion": "2",
          "pypiPackageName": "test-sdk",
          "githubRepoName": "python-sdk",
          "sdkVersion": "1.0.0"
        },
        "typescript": {
          "npmName": "test-sdk",
          "npmOrg": "myorg",
          "githubRepoName": "typescript-sdk",
          "sdkVersion": "1.0.0"
        },
        "java": {
          "groupId": "com.myorg",
          "artifactId": "test-sdk",
          "homepage": "https://myorg.com",
          "githubRepoName": "java-sdk",
          "sdkVersion": "1.0.0",
          "developers": [
            {
              "name": "John Doe",
              "email": "[email protected]",
              "organization": "My Organization",
              "organizationUrl": "https://myorg.com"
            }
          ]
        },
        "go": {
          "goModuleName": "github.com/myorg/go-sdk",
          "githubRepoName": "go-sdk",
          "sdkVersion": "1.0.0",
        },
        "php": {
          "packageName": "myorg/test-sdk",
          "githubRepoName": "php-sdk",
          "sdkVersion": "1.0.0"
        },
      },
      "publishing": {
        "githubOrg": "myorg"
      }
    }

  5. Commit your changes to the liblab.config.json and API spec files.

  6. Push the changes to your control repo. This will trigger the Jenkins pipeline that generates your SDKs and creates pull requests in your SDK repos.

7. Check the Jenkins pipeline​

After pushing your changes to the control repo, Jenkins will automatically scan the repo. If it detects changes, it will trigger a new build for the pipeline.

Triggering a New Build

If you want to manually force Jenkins to scan your repo for changes, go to the pipeline you created in Step 5 and click Scan Repository Now.

Scan repo now

To review the results of the latest Jenkins pipeline job, follow the steps:

  1. Navigate to the pipeline you created in Step 5.
  2. Click Build History to view the results of previous builds.
  3. Select the build number of the most recent job to open its summary.
  4. Review the Pipeline Overview to inspect each pipeline stage. If errors arise, you can check the output of each stage to identify the problem.

The following GIF demonstrates how to inspect the results of your builds.

Sucessfull pipeline job gif

8. Merge SDK PRs​

After Jenkins completes the pipeline, a pull request (PR) will be created automatically in each SDK repo by liblab.

To view the PRs, access each SDK repo on GitHub and navigate to the Pull Requests section to view the new PRs created by the liblab CLI.

New PR created by liblab CLI

note

If you don't see the PRs, review Jenkins' build history to check for any errors during the SDK generation process.

To finish, follow the steps:

  1. Open each PR to review the changes and ensure the SDK has been generated correctly.
  2. Verify that the modifications align with your expectations and meet project requirements.
  3. If everything is correct, approve the PR and merge it into the main branch.

9. Create a release in GitHub​

You need to create a release in the SDK repo to trigger the GitHub Action that publishes your SDK to the package manager. Releases must include a tag that specifies the commit you want to release.

Follow the steps to configure your SDKs' repos to create releases:

  1. From each SDK repo, open the SDK repo on GitHub. Click Create a new release from the sidebar Releases section.

    The releases section

  2. Click the Choose a tag dropdown, enter a tag name, and select Create new tag. You must use semantic versioning for the tag, such as v1.0.0. It's common to prefix the version with v.

Versioning for Go Packages

The release tag will be used as the version number in the Go Packages repo. Ensure the tag follows semantic versioning and matches the version specified in the liblab.config.json file.

  1. Enter a title for the release, which can match the tag name, and add a description to summarize the changes or improvements in the release.

  2. Click Publish release to finalize it.

Publishing the release will automatically trigger the GitHub Action to publish the SDK to the relevant package manager.

10. Confirm SDK publication​

After completing the publishing process, check the package manager services related to the SDKs you have created, such as npm, PyPI, and Maven, to verify that your SDK has been successfully published.

🎉🎉🎉   Congratulations! You have successfully generated and published your SDK!   🎉🎉🎉

Your setup is now fully automated to generate and publish SDKs with liblab and Jenkins pipelines, ensuring your latest API updates are quickly available to your users.

If you don’t see the SDK in the package manager, check the CI/CD > Pipelines or Jobs section in each SDK repo to troubleshoot any errors during the publishing process.

Last updated on