Add support for structured logs

Dockerfile

@@ -42,11 +42,13 @@ COPY ./packages/db ./packages/db
 COPY ./packages/schemas ./packages/schemas
 COPY ./packages/crypto ./packages/crypto
 COPY ./packages/error ./packages/error
+COPY ./packages/logger ./packages/logger
 
 RUN yarn workspace @sourcebot/db install
 RUN yarn workspace @sourcebot/schemas install
 RUN yarn workspace @sourcebot/crypto install
 RUN yarn workspace @sourcebot/error install
+RUN yarn workspace @sourcebot/logger install
 # ------------------------------------
 
 # ------ Build Web ------
@@ -89,6 +91,7 @@ COPY --from=shared-libs-builder /app/packages/db ./packages/db
 COPY --from=shared-libs-builder /app/packages/schemas ./packages/schemas
 COPY --from=shared-libs-builder /app/packages/crypto ./packages/crypto
 COPY --from=shared-libs-builder /app/packages/error ./packages/error
+COPY --from=shared-libs-builder /app/packages/logger ./packages/logger
 
 # Fixes arm64 timeouts
 RUN yarn workspace @sourcebot/web install
@@ -128,6 +131,7 @@ COPY --from=shared-libs-builder /app/packages/db ./packages/db
 COPY --from=shared-libs-builder /app/packages/schemas ./packages/schemas
 COPY --from=shared-libs-builder /app/packages/crypto ./packages/crypto
 COPY --from=shared-libs-builder /app/packages/error ./packages/error
+COPY --from=shared-libs-builder /app/packages/logger ./packages/logger
 RUN yarn workspace @sourcebot/backend install
 RUN yarn workspace @sourcebot/backend build
 
@@ -209,6 +213,7 @@ COPY --from=shared-libs-builder /app/packages/db ./packages/db
 COPY --from=shared-libs-builder /app/packages/schemas ./packages/schemas
 COPY --from=shared-libs-builder /app/packages/crypto ./packages/crypto
 COPY --from=shared-libs-builder /app/packages/error ./packages/error
+COPY --from=shared-libs-builder /app/packages/logger ./packages/logger
 
 # Configure dependencies
 RUN apk add --no-cache git ca-certificates bind-tools tini jansson wget supervisor uuidgen curl perl jq redis postgresql postgresql-contrib openssl util-linux unzip

docs/docs.json

@@ -74,7 +74,8 @@
                                     "docs/configuration/auth/roles-and-permissions"
                                 ]
                             },
-                            "docs/configuration/transactional-emails"
+                            "docs/configuration/transactional-emails",
+                            "docs/configuration/structured-logging"
                         ]
                     },
                     {

docs/docs/configuration/auth/overview.mdx

@@ -2,6 +2,8 @@
 title: Overview
 ---
 
+<Warning>If you're deploying Sourcebot behind a domain, you must set the [AUTH_URL](/docs/configuration/environment-variables) environment variable.</Warning>
+
 Sourcebot has built-in authentication that gates access to your organization. OAuth, email codes, and email / password are supported. 
 
 The first account that's registered on a Sourcebot deployment is made the owner. All other users who register must be [approved](/docs/configuration/auth/overview#approving-new-members) by the owner.
@@ -40,8 +42,6 @@ See [transactional emails](/docs/configuration/transactional-emails) for more de
 
 ## Enterprise Authentication Providers
 
-<Warning>If you're deploying Sourcebot behind a domain, you must set the [AUTH_URL](/docs/configuration/environment-variables) environment variable to use these providers.</Warning>
-
 The following authentication providers require an [enterprise license](/docs/license-key) to be enabled.
 
 By default, a new user registering using these providers must have their join request accepted by the owner of the organization to join. To allow a user to join automatically when

docs/docs/configuration/auth/roles-and-permissions.mdx

@@ -1,5 +1,6 @@
 ---
 title: Roles and Permissions
+sidebarTitle: Roles and permissions
 ---
 
 <Note>Looking to sync permissions with your identify provider? We're working on it - [reach out](https://www.sourcebot.dev/contact) to us to learn more</Note>

docs/docs/configuration/environment-variables.mdx

@@ -27,6 +27,8 @@ The following environment variables allow you to configure your Sourcebot deploy
 | `SMTP_CONNECTION_URL` | `-` | <p>The url to the SMTP service used for sending transactional emails. See [this doc](/docs/configuration/transactional-emails) for more info.</p> |
 | `SOURCEBOT_ENCRYPTION_KEY` | Automatically generated at startup if no value is provided. Generated using `openssl rand -base64 24` | <p>Used to encrypt connection secrets and generate API keys.</p> |
 | `SOURCEBOT_LOG_LEVEL` | `info` | <p>The Sourcebot logging level. Valid values are `debug`, `info`, `warn`, `error`, in order of severity.</p> |
+| `SOURCEBOT_STRUCTURED_LOGGING_ENABLED` | `false` | <p>Enables/disable structured JSON logging. See [this doc](/docs/configuration/structured-logging) for more info.</p> |
+| `SOURCEBOT_STRUCTURED_LOGGING_FILE` | - | <p>Optional file to log to if structured logging is enabled</p> | 
 | `SOURCEBOT_TELEMETRY_DISABLED` | `false` | <p>Enables/disables telemetry collection in Sourcebot. See [this doc](/docs/overview.mdx#telemetry) for more info.</p> |
 | `TOTAL_MAX_MATCH_COUNT` | `100000` | <p>The maximum number of matches per query</p> |
 | `ZOEKT_MAX_WALL_TIME_MS` | `10000` | <p>The maximum real world duration (in milliseconds) per zoekt query</p> | 

docs/docs/configuration/structured-logging.mdx

@@ -0,0 +1,39 @@
+---
+title: Structured logging
+---
+
+By default, Sourcebot will output logs to the console in a human readable format. If you'd like Sourcebot to output structured JSON logs, set the following env vars:
+
+- `SOURCEBOT_STRUCTURED_LOGGING_ENABLED` (default: `false`): Controls whether logs are in a structured JSON format
+- `SOURCEBOT_STRUCTURED_LOGGING_FILE`: If structured logging is enabled and this env var is set, structured logs will be written to this file (ex. `/data/sourcebot.log`)
+
+### Structured log schema
+```json
+{
+    "$schema": "http://json-schema.org/draft-07/schema#",
+    "type": "object",
+    "title": "SourcebotLog",
+    "properties": {
+        "level": {
+            "type": "string",
+            "description": "The log level (error, warning, info, debug)"
+        },
+        "service": {
+            "type": "string",
+            "description": "The Sourcebot component that generated the log"
+        },
+        "message": {
+            "type": "string",
+            "description": "The log message"
+        },
+        "status": {
+            "type": "string",
+            "description": "The same value as the level field added for datadog support"
+        },
+        "timestamp": {
+            "type": "string",
+            "description": "The timestamp of the log in ISO 8061 format"
+        }
+    }
+}
+```

docs/docs/connections/overview.mdx

@@ -43,7 +43,7 @@ A JSON configuration file is used to specify connections. For example:
 
 Configuration files must conform to the [JSON schema](#schema-reference).
 
-When running Sourcebot, this file must be mounted in a volume that is accessible to the container, with it's path specified in the `CONFIG_PATH` environment variable. For example:
+When running Sourcebot, this file must be mounted in a volume that is accessible to the container, with its path specified in the `CONFIG_PATH` environment variable. For example:
 
 ```bash
 docker run \

docs/docs/deployment-guide.mdx

@@ -53,6 +53,9 @@ Watch this 1:51 minute video to get a quick overview of how to deploy Sourcebot
     </Step>
 
     <Step title="Launch your instance">
+        <Warning>If you're deploying Sourcebot behind a domain, you must set the [AUTH_URL](/docs/configuration/environment-variables) environment variable.</Warning>
+
+
         In the same directory as `config.json`, run the following command to start your instance:
 
         ``` bash

docs/docs/features/agents/review-agent.mdx

@@ -1,6 +1,6 @@
 ---
 title: AI Code Review Agent
-sidebarTitle: AI Code Review Agent
+sidebarTitle: AI code review agent
 ---
 
 <Note>

packages/backend/package.json

@@ -23,15 +23,14 @@
     },
     "dependencies": {
         "@gitbeaker/rest": "^40.5.1",
-        "@logtail/node": "^0.5.2",
-        "@logtail/winston": "^0.5.2",
         "@octokit/rest": "^21.0.2",
         "@sentry/cli": "^2.42.2",
         "@sentry/node": "^9.3.0",
         "@sentry/profiling-node": "^9.3.0",
         "@sourcebot/crypto": "workspace:*",
         "@sourcebot/db": "workspace:*",
         "@sourcebot/error": "workspace:*",
+        "@sourcebot/logger": "workspace:*",
         "@sourcebot/schemas": "workspace:*",
         "@t3-oss/env-core": "^0.12.0",
         "@types/express": "^5.0.0",
@@ -51,7 +50,6 @@
         "prom-client": "^15.1.3",
         "simple-git": "^3.27.0",
         "strip-json-comments": "^5.0.1",
-        "winston": "^3.15.0",
         "zod": "^3.24.3"
     }
 }

packages/backend/src/bitbucket.ts

@@ -2,7 +2,7 @@ import { createBitbucketCloudClient } from "@coderabbitai/bitbucket/cloud";
 import { createBitbucketServerClient } from "@coderabbitai/bitbucket/server";
 import { BitbucketConnectionConfig } from "@sourcebot/schemas/v3/bitbucket.type";
 import type { ClientOptions, ClientPathsWithMethod } from "openapi-fetch";
-import { createLogger } from "./logger.js";
+import { createLogger } from "@sourcebot/logger";
 import { PrismaClient } from "@sourcebot/db";
 import { getTokenFromConfig, measure, fetchWithRetry } from "./utils.js";
 import * as Sentry from "@sentry/node";
@@ -13,7 +13,7 @@ import { SchemaRestRepository as ServerRepository } from "@coderabbitai/bitbucke
 import { processPromiseResults } from "./connectionUtils.js";
 import { throwIfAnyFailed } from "./connectionUtils.js";
 
-const logger = createLogger("Bitbucket");
+const logger = createLogger('bitbucket');
 const BITBUCKET_CLOUD_GIT = 'https://bitbucket.org';
 const BITBUCKET_CLOUD_API = 'https://api.bitbucket.org/2.0';
 const BITBUCKET_CLOUD = "cloud";

packages/backend/src/connectionManager.ts

@@ -2,7 +2,7 @@ import { Connection, ConnectionSyncStatus, PrismaClient, Prisma } from "@sourceb
 import { Job, Queue, Worker } from 'bullmq';
 import { Settings } from "./types.js";
 import { ConnectionConfig } from "@sourcebot/schemas/v3/connection.type";
-import { createLogger } from "./logger.js";
+import { createLogger } from "@sourcebot/logger";
 import { Redis } from 'ioredis';
 import { RepoData, compileGithubConfig, compileGitlabConfig, compileGiteaConfig, compileGerritConfig, compileBitbucketConfig, compileGenericGitHostConfig } from "./repoCompileUtils.js";
 import { BackendError, BackendException } from "@sourcebot/error";
@@ -32,7 +32,7 @@ type JobResult = {
 export class ConnectionManager implements IConnectionManager {
     private worker: Worker;
     private queue: Queue<JobPayload>;
-    private logger = createLogger('ConnectionManager');
+    private logger = createLogger('connection-manager');
 
     constructor(
         private db: PrismaClient,

packages/backend/src/env.ts

@@ -22,7 +22,6 @@ dotenv.config({
 export const env = createEnv({
     server: {
         SOURCEBOT_ENCRYPTION_KEY: z.string(),
-        SOURCEBOT_LOG_LEVEL: z.enum(["info", "debug", "warn", "error"]).default("info"),
         SOURCEBOT_TELEMETRY_DISABLED: booleanSchema.default("false"),
         SOURCEBOT_INSTALL_ID: z.string().default("unknown"),
         NEXT_PUBLIC_SOURCEBOT_VERSION: z.string().default("unknown"),

packages/backend/src/gerrit.ts

@@ -1,6 +1,6 @@
 import fetch from 'cross-fetch';
 import { GerritConnectionConfig } from "@sourcebot/schemas/v3/index.type"
-import { createLogger } from './logger.js';
+import { createLogger } from '@sourcebot/logger';
 import micromatch from "micromatch";
 import { measure, fetchWithRetry } from './utils.js';
 import { BackendError } from '@sourcebot/error';
@@ -33,7 +33,7 @@ interface GerritWebLink {
    url: string;
 }
 
-const logger = createLogger('Gerrit');
+const logger = createLogger('gerrit');
 
 export const getGerritReposFromConfig = async (config: GerritConnectionConfig): Promise<GerritProject[]> => {
    const url = config.url.endsWith('/') ? config.url : `${config.url}/`;
@@ -95,7 +95,7 @@ const fetchAllProjects = async (url: string): Promise<GerritProject[]> => {
       try {
          response = await fetch(endpointWithParams);
          if (!response.ok) {
-            console.log(`Failed to fetch projects from Gerrit at ${endpointWithParams} with status ${response.status}`);
+            logger.error(`Failed to fetch projects from Gerrit at ${endpointWithParams} with status ${response.status}`);
             const e = new BackendException(BackendError.CONNECTION_SYNC_FAILED_TO_FETCH_GERRIT_PROJECTS, {
                status: response.status,
             });
@@ -109,7 +109,7 @@ const fetchAllProjects = async (url: string): Promise<GerritProject[]> => {
          }
 
          const status = (err as any).code;
-         console.log(`Failed to fetch projects from Gerrit at ${endpointWithParams} with status ${status}`);
+         logger.error(`Failed to fetch projects from Gerrit at ${endpointWithParams} with status ${status}`);
          throw new BackendException(BackendError.CONNECTION_SYNC_FAILED_TO_FETCH_GERRIT_PROJECTS, {
             status: status,
          });

packages/backend/src/gitea.ts

@@ -2,14 +2,14 @@ import { Api, giteaApi, HttpResponse, Repository as GiteaRepository } from 'gite
 import { GiteaConnectionConfig } from '@sourcebot/schemas/v3/gitea.type';
 import { getTokenFromConfig, measure } from './utils.js';
 import fetch from 'cross-fetch';
-import { createLogger } from './logger.js';
+import { createLogger } from '@sourcebot/logger';
 import micromatch from 'micromatch';
 import { PrismaClient } from '@sourcebot/db';
 import { processPromiseResults, throwIfAnyFailed } from './connectionUtils.js';
 import * as Sentry from "@sentry/node";
 import { env } from './env.js';
 
-const logger = createLogger('Gitea');
+const logger = createLogger('gitea');
 const GITEA_CLOUD_HOSTNAME = "gitea.com";
 
 export const getGiteaReposFromConfig = async (config: GiteaConnectionConfig, orgId: number, db: PrismaClient) => {

packages/backend/src/github.ts

@@ -1,6 +1,6 @@
 import { Octokit } from "@octokit/rest";
 import { GithubConnectionConfig } from "@sourcebot/schemas/v3/github.type";
-import { createLogger } from "./logger.js";
+import { createLogger } from "@sourcebot/logger";
 import { getTokenFromConfig, measure, fetchWithRetry } from "./utils.js";
 import micromatch from "micromatch";
 import { PrismaClient } from "@sourcebot/db";
@@ -9,7 +9,7 @@ import { processPromiseResults, throwIfAnyFailed } from "./connectionUtils.js";
 import * as Sentry from "@sentry/node";
 import { env } from "./env.js";
 
-const logger = createLogger("GitHub");
+const logger = createLogger('github');
 const GITHUB_CLOUD_HOSTNAME = "github.com";
 
 export type OctokitRepository = {

packages/backend/src/gitlab.ts

@@ -1,14 +1,14 @@
 import { Gitlab, ProjectSchema } from "@gitbeaker/rest";
 import micromatch from "micromatch";
-import { createLogger } from "./logger.js";
+import { createLogger } from "@sourcebot/logger";
 import { GitlabConnectionConfig } from "@sourcebot/schemas/v3/gitlab.type"
 import { getTokenFromConfig, measure, fetchWithRetry } from "./utils.js";
 import { PrismaClient } from "@sourcebot/db";
 import { processPromiseResults, throwIfAnyFailed } from "./connectionUtils.js";
 import * as Sentry from "@sentry/node";
 import { env } from "./env.js";
 
-const logger = createLogger("GitLab");
+const logger = createLogger('gitlab');
 export const GITLAB_CLOUD_HOSTNAME = "gitlab.com";
 
 export const getGitLabReposFromConfig = async (config: GitlabConnectionConfig, orgId: number, db: PrismaClient) => {

packages/backend/src/index.ts

@@ -8,31 +8,34 @@ import { AppContext } from "./types.js";
 import { main } from "./main.js"
 import { PrismaClient } from "@sourcebot/db";
 import { env } from "./env.js";
+import { createLogger } from "@sourcebot/logger";
+
+const logger = createLogger('index');
 
 // Register handler for normal exit
 process.on('exit', (code) => {
-    console.log(`Process is exiting with code: ${code}`);
+    logger.info(`Process is exiting with code: ${code}`);
 });
 
 // Register handlers for abnormal terminations
 process.on('SIGINT', () => {
-    console.log('Process interrupted (SIGINT)');
-    process.exit(130);
+    logger.info('Process interrupted (SIGINT)');
+    process.exit(0);
 });
 
 process.on('SIGTERM', () => {
-    console.log('Process terminated (SIGTERM)');
-    process.exit(143);
+    logger.info('Process terminated (SIGTERM)');
+    process.exit(0);
 });
 
 // Register handlers for uncaught exceptions and unhandled rejections
 process.on('uncaughtException', (err) => {
-    console.log(`Uncaught exception: ${err.message}`);
+    logger.error(`Uncaught exception: ${err.message}`);
     process.exit(1);
 });
 
 process.on('unhandledRejection', (reason, promise) => {
-    console.log(`Unhandled rejection at: ${promise}, reason: ${reason}`);
+    logger.error(`Unhandled rejection at: ${promise}, reason: ${reason}`);
     process.exit(1);
 });
 
@@ -60,12 +63,12 @@ main(prisma, context)
         await prisma.$disconnect();
     })
     .catch(async (e) => {
-        console.error(e);
+        logger.error(e);
         Sentry.captureException(e);
 
         await prisma.$disconnect();
         process.exit(1);
     })
     .finally(() => {
-        console.log("Shutting down...");
+        logger.info("Shutting down...");
     });

packages/backend/src/instrument.ts

@@ -1,5 +1,8 @@
 import * as Sentry from "@sentry/node";
 import { env } from "./env.js";
+import { createLogger } from "@sourcebot/logger";
+
+const logger = createLogger('instrument');
 
 if (!!env.NEXT_PUBLIC_SENTRY_BACKEND_DSN && !!env.NEXT_PUBLIC_SENTRY_ENVIRONMENT) {
     Sentry.init({
@@ -8,5 +11,5 @@ if (!!env.NEXT_PUBLIC_SENTRY_BACKEND_DSN && !!env.NEXT_PUBLIC_SENTRY_ENVIRONMENT
         environment: env.NEXT_PUBLIC_SENTRY_ENVIRONMENT,
     });
 } else {
-    console.debug("Sentry was not initialized");
+    logger.debug("Sentry was not initialized");
 }