Set Up a Datadog Agent in Railway
Datadog provides a centralized location for logs, metrics, and traces emitted from applications deployed in various locations.
While Railway has a native, centralized logging mechanism, you may have a need to ship this data to another location, to view it alongside data collected from systems outside of Railway.
Objectives
In this tutorial you will learn how to -
- Deploy a Datadog agent in Railway - listening for metrics, logs, and traces.
- Configure an application to send metrics, logs, and traces to the agent.
If you are looking for a quicker way to get started, you can also deploy this project from a template.
Prerequisites
To be successfull, you should already have -
- Railway CLI installed
- Datadog API key and site value
Caveats
Keep in mind that the Datadog agent sends data to Datadog over the Internet, meaning you will see an increase in egress cost. If this is a concern, you may be interested in exploring self-hosted solutions, and we encourage you to check out the OpenTelemetry Tutorial.
1. Create the Project Structure
First we'll create the project structure.
From your local machine -
- Create a folder for your project called
railway-project
. - Create two folders inside of
railway-project
calledagent
andexpressapi
.
You project structure should look like this
2. Set Up the Datadog Agent
Now we'll add files to the agent
folder, which will build the Datadog Agent image.
- Inside of the
agent
folder, create three files -Dockerfile
syslog.yaml
datadog.yaml
Define the Dockerfile
Let's define the Dockerfile.
-
Within your Dockerfile, add the following contents.
Define the syslog.yaml file
The syslog.yaml
file is used to instruct the agent to listen for syslogs to be forwarded on the configured port.
-
Within the
syslog.yaml
file, add the following contents -
Define the datadog.yaml file
The datadog.yaml
file is used to instruct the agent to send logs to Datadog over http
instead of the default tcp
.
-
Within the
datadog.yaml
file, add the following contents -
3. Set Up the Node Express App
Now let's build a Node Express App that will send logs and metrics to the Datadog Agent over the Private Network.
-
Create an
app.js
file inside of theexpressapi
folder you created in Step 1. -
Use
npm
(or your preferred package manager) to install the required dependencies -
Define the app.js file
The app.js
file defines your express server. This is where we will import the DataDog tracer and initialize the StatsD client and the Winston logger, which will send traces, metrics, and logs, respectively, to the Datadog agent.
-
Within the
app.js
file, add the following contents -
Winston and hot-shots
In this example app, we are using Winston
as the logger and hot-shots
as the StatsD client.
Winston
is configured usingwinston-syslog
to transport logs to the Datadog agent via Syslog overudp6
.hot-shots
is configured to send metrics to the Datadog agent overudp6
.
4. Set Up the Railway Project
Now let's create the project using the CLI, then create the services and variables from within the project in Railway.
You will need your Datadog API key and Site value in this step.
If you have not already done so, please install the CLI and authenticate.
Create a Project
-
In your terminal, run the following command to create a new project -
-
Name your project
datadog-project
when prompted (you can change this later). -
Open your project in Railway by running the following -
Create the Services
- In Railway, create an Empty Service by clicking
+ New
button in the top right-hand corner and choosingEmpty Service
in the prompt. - Right click on the service that is created, select
Update Info
and name itdatadog-agent
. - Repeat the above steps to create a second service, but name the second service
expressapi
.
Add the Variables
Each service requires unique variables listed below. For each service, follow the steps to add the variables required for the service.
Variables -
datadog-agent
variables
expressapi
variables
- Click on the service card
- Click on the
Variables
tab - Click on
Raw Editor
- Paste the required variables (be sure to update the Datadog API key and site with your own values)
- Click
Update Variables
andDeploy
5. Deploy to Railway
Now we're ready to deploy our services. We will use the CLI to push code from our local machine to Railway.
Railway Up
Follow these steps for each service -
- In your local terminal, change directory into the
agent
folder. - Link to
datadog-project
by running the following command - - Follow the prompts, selecting the
datadog-project
andproduction
environment. - Link to the
datadog-agent
service by running the following command - - Follow the prompt, selecting the
datadog-agent
service. - Deploy the agent by running the following command -
- Change directory into your
expressapi
folder and repeat the steps above, but for theexpressapi
service.
Create a Domain for the Express App
The express app will send logs and metrics to the Datadog agent upon navigation to either of its two routes. So let's give it a domain -
- Ensure that you are linked to the
datadog-project
andexpressapi
service (refer to the steps above) - Assign the
expressapi
a domain by running the following command -
6. Test and Confirm
Test that your Datadog Agent is receiving and forwarding data to Datadog by navigating to the routes in the Express app -
/
/test
Generate some traffic to these two routes and verify in your Datadog instance that the data is there.
Note: it can take a few minutes to see the data in Datadog, check the Datadog Agent's logs in Railway
Conclusion
Congratulations! You have deployed a Datadog Agent and a Node Express app that sends logs and metrics to Datadog.
This is a very basic implementation, and you should refer to the Datadog documentation for information on how to customize the data you send.
Edit this file on GitHub