Deploy Gemma3 through the Linode Marketplace

Google Gemma 3 is an open-source large language model from Google DeepMind, built with the same technology that powers Google’s Gemini models. This Marketplace App deploys Gemma 3 with Open WebUI, a self-hosted web interface for interacting with LLMs. The deployment uses GPU-accelerated inference and lets you choose between two model sizes: 4B for efficient performance or 12B for enhanced capabilities.

Deploying a Marketplace App

The Linode Marketplace lets you easily deploy software on a Compute Instance using Cloud Manager. See Get Started with Marketplace Apps for complete steps.

1. Log in to Cloud Manager and select the Marketplace link from the left navigation menu. This displays the Linode Create page with the Marketplace tab pre-selected.

2. Under the Select App section, select the app you would like to deploy.

3. Complete the form by following the steps and advice within the Creating a Compute Instance guide. Depending on the Marketplace App you selected, there may be additional configuration options available. See the Configuration Options section below for compatible distributions, recommended plans, and any additional configuration options available for this Marketplace App.

4. Click the Create Linode button. Once the Compute Instance has been provisioned and has fully powered on, wait for the software installation to complete. If the instance is powered off or restarted before this time, the software installation will likely fail.

To verify that the app has been fully installed, see Get Started with Marketplace Apps > Verify Installation. Once installed, follow the instructions within the Getting Started After Deployment section to access the application and start using it.

Configuration Options

• Supported distributions: Ubuntu 24.04 LTS
• Recommended plan: All GPU plan types and sizes can be used. The 4B model requires at least 16GB of RAM, while the 12B model requires at least 32GB of RAM.

Gemma3 Options

• Admin Login Name (required): The initial admin username for accessing the Open WebUI interface.
• Admin Login Email (required): The email address for the Open WebUI admin account.
• Hugging Face API Token (required): A Hugging Face API token is required to download the Gemma 3 model. See Obtaining a Hugging Face Token below for instructions.
• Model Size (required): Choose between 4B (16GB+ RAM required) or 12B (32GB+ RAM required).

Obtaining a Hugging Face Token

Before deployment, you need a Hugging Face API token to access the Gemma 3 model. To obtain one:

1. Create a free account at huggingface.co/join.
2. Accept the Gemma license at huggingface.co/google/gemma-3-12b-it.
3. Generate a token at huggingface.co/settings/tokens. Read-only access is sufficient.
4. Provide this token during the Marketplace deployment process.

Limited Sudo User

You need to fill out the following fields to automatically create a limited sudo user, with a strong generated password for your new Compute Instance. This account will be assigned to the sudo group, which provides elevated permissions when running commands with the sudo prefix.

• Limited sudo user: Enter your preferred username for the limited user. No Capital Letters, Spaces, or Special Characters.

  Locating The Generated Sudo Password

  A password is generated for the limited user and stored in a .credentials file in their home directory, along with application specific passwords. This can be viewed by running: cat /home/$USERNAME/.credentials

  For best results, add an account SSH key for the Cloud Manager user that is deploying the instance, and select that user as an authorized_user in the API or by selecting that option in Cloud Manager. Their SSH pubkey will be assigned to both root and the limited user.

• Disable root access over SSH: To block the root user from logging in over SSH, select Yes. You can still switch to the root user once logged in, and you can also log in as root through Lish.

  Accessing The Instance Without SSH

  If you disable root access for your deployment and do not provide a valid Account SSH Key assigned to the authorized_user, you will need to login as the root user via the Lish console and run cat /home/$USERNAME/.credentials to view the generated password for the limited user.

Custom Domain (Optional)

If you wish to automatically configure a custom domain, you first need to configure your domain to use Linode’s name servers. This is typically accomplished directly through your registrar. See Use Linode’s Name Servers with Your Domain. Once that is finished, you can fill out the following fields for the Marketplace App:

• Linode API Token: If you wish to use the Linode’s DNS Manager to manage DNS records for your custom domain, create a Linode API Personal Access Token on your account with Read/Write access to Domains. If this is provided along with the subdomain and domain fields (outlined below), the installation attempts to create DNS records via the Linode API. See Get an API Access Token. If you do not provide this field, you need to manually configure your DNS records through your DNS provider and point them to the IP address of the new instance.
• Subdomain: The subdomain you wish to use, such as www for www.example.com.
• Domain: The domain name you wish to use, such as example.com.

Getting Started after Deployment

Access Your Credentials

Your Open WebUI admin credentials are stored in a .credentials file on your instance. To retrieve them:

1. Log in to your instance via SSH or Lish. See Connecting to a Remote Server Over SSH for assistance, or use the Lish Console.

2. Once logged in, retrieve your credentials from the .credentials file:

   sudo cat /home/$USER/.credentials

   The .credentials file contains:

   • Sudo Username and Password: Your limited sudo user credentials
   • Open WebUI Admin Name: The admin username for the web interface
   • Open WebUI Admin Email: The admin email address
   • Open WebUI Admin Password: The admin password for logging in

Access the Open WebUI Interface

Once your instance has finished deploying, you can access the Open WebUI interface through your browser:

1. Navigate to your instance’s domain or rDNS address via HTTPS (e.g., https://your-domain.com or https://192-0-2-1.ip.linodeusercontent.com).

2. Log in using the admin email and password from the .credentials file.

3. You can now start chatting with the Gemma 3 model.

Working with RAG Operations

Open WebUI provides built-in support for RAG (Retrieval Augmented Generation) operations, allowing you to upload documents and chat with them. By default, the deployment supports file uploads up to 100MB.

If you need to upload larger documents, you can modify the NGINX configuration:

1. Edit the NGINX virtual host configuration file located at /etc/nginx/sites-enabled/your-domain.

2. Update the client_max_body_size directive to a larger value (e.g., client_max_body_size 500M; for 500MB).

3. Reload NGINX:

   sudo systemctl reload nginx

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

• Gemma3 Official Documentation
• Google Gemma 3 4B Model
• Google Gemma 3 12B Model
• Open WebUI Documentation