Client, backend, and services for Dark https://darklang.com
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
 
 
Stachu Korick 5a2e464820
Merge pull request #4227 from darklang/StachuDotNet-patch-1
4 days ago
.circleci Merge branch 'main' into remove-ocaml-analysis 2 weeks ago
.devcontainer Remove some volumes that are no longer used 4 weeks ago
.github Remove stroller and rust compilation machinery 1 month ago
.vscode We don't use ApiServer2 anymore 5 months ago
auth0-branding Put auth0 branding into static assets 2 years ago
backend Formatting 4 days ago
client Remove mID from the definition of patterns 5 days ago
config Remove old env vars only used for f#/ocaml communication 4 days ago
containers Don't need to check old binary 1 month ago
docs Clarify usage of some of our production accounts in docs 6 days ago
esy.lock Update to OCaml 4.07.1 2 years ago
fsharp-backend Remove console log 4 days ago
integration-tests Reference window.Dark.analysis over window.Dark.fsharpAnalysis 2 weeks ago
rundir Move runtime files, etc, into a separate directory outside server. 4 years ago
scripts Remove unused ocaml that's easy to remove 1 week ago
services Remove stroller and rust compilation machinery 1 month ago
.dockerignore None of these files should be in dockerignore - most should be in volumes 2 years ago
.editorconfig Don't force test files to have a newline at the end 8 months ago
.gitattributes Try crlf attributes for paket.restore.targets 2 years ago
.gitignore Remove stroller and rust compilation machinery 1 month ago
.ocamlformat ocamlformat: Allow single blank lines to separate code blocks 2 years ago
.prettierignore No longer need to ignore BlazorWorker in prettier 12 months ago
.prettierrc.toml Regenerate prettier for new version (using width of 80 to match ocaml) 2 years ago
.style.yapf Add formatting to python files 10 months ago
.testcaferc.json Move testcafe settings into a config file 3 years ago
.yamllint Lint .circleci/config.yml too 1 year ago
CHANGELOG.md Link to the changelog 2 years ago
CODE-OF-CONDUCT.md Clarify enforcement 6 months ago
CODING-GUIDE.md Don't error when the push is too big 1 month ago
Dockerfile Move git-restore-mtime to the dockerfile 4 weeks ago
LICENSE.md Fix formatting 2 years ago
LICENSES Remove remnants of postgres-honeytail 1 month ago
README.md Merge branch 'main' into remove-ocaml-analysis 2 weeks ago
bsconfig.json Remove unused key 1 week ago
current-cluster Don't hard-code 'darkcluster1' everywhere. 4 years ago
dune Update to the latest version of tablecloth 11 months ago
dune-project move bsb & esy builds to app root 2 years ago
esy.json Update to OCaml 4.07.1 2 years ago
fsharplint.json Remove some lint suggestions 11 months ago
package-lock.json Bump shell-quote from 1.7.2 to 1.7.3 1 week ago
package.json Merge pull request #4177 from darklang/dependabot/npm_and_yarn/jsdom-20.0.0 1 week ago
rustfmt.toml Add support for formatting rust 2 years ago

README.md

Dark

This is the main repo for Dark, a combined language, editor, and infrastructure to make it easy to build backends.

This repo is intended to help Dark users solve their needs by fixing bugs, expanding features, or otherwise contributing. Dark is source available, not open source.

See also:

See our guide to the repo for help browsing.

Contributing

We are committed to make Dark easy to contribute to. Our contributor docs will help guide you through your first PR, find good projects to contribute to, and learn about the code base.

Getting started

We try to make it really easy to get started. If you have any problems, please ask in Slack and we'll work to fix any issues you have.

Install dependencies

We develop Dark within a docker container, so there is not a lot of setup. However, we do need to setup the host system in a few ways to support running scripts, and Docker.

OSX

To build and run the server you must have the following installed (and running):

Linux

Everything should just work on Linux, so long as you have docker installed and you are using bash 4 or later.

Windows

On Windows, you can run Dark in WSL2 (Windows Subsystem for Linux):

  • You must be on at least Windows 10 Version 2004, and you must run WSL 2 (docker does not work in WSL 1)
  • Follow the WSL 2 installation instructions
  • Follow the Docker for WSL 2 installation instructions
  • You need to clone the dark repo with the git core.autocrlf setting set to false. You can configure this by running git config --global core.autocrlf false. If you have already cloned dark, you will need to reclone it.
  • This section of the guide is incomplete. Please create an issue if you find something doesn't work.

Building and running for the first time

Running the build script

Now that the pre-requisites are installed, we should be able to build the development container in Docker, which has the exact right versions of all the tools we use.

  • If you're using VSCode, we run our build scripts in the VSCode devcontainer. See the VSCode instructions for instructions.
  • Otherwise, simply run scripts/builder --compile --watch --test

These steps apply for all builds, VSCode or using scripts/builder:

  • Wait until the terminal says "Initial compile succeeded" - this means the build server is ready. The builder script will sit open, waiting for file changes in order to recompile
  • If you see "initial compile failed", it may be a memory issue. Sometimes trying again will work. If not, ensure you have Docker configured to provide 4GB or more of memory, then try again.
  • Open your browser to http://darklang.localhost:9000/a/dark/, username "dark", password "what"
  • Edit code normally - on each save to your filesystem, the app will be rebuilt and the browser will reload as necessary

Using Dark scripts

The scripts/ directory is full of scripts. They automatically execute in the dev container, even if they are run on the host (see scripts/devcontainer/_assert-in-container for how this works). Scripts starting with an underscore are primarily intended to be run by other scripts. Scripts without an underscore are usually intended to be called by a human, though they are often also called by other scripts as well.

Read the contributor docs

If you've gotten this far, you're now ready to contribute your first PR.

Advanced setup

Testing

Unit tests run when you specify --test to scripts/builder. You can run them as a once off using:

  • scripts/run-client-tests
  • scripts/run-backend-tests
  • scripts/run-fsharp-tests

Integration tests:

  • scripts/run-in-docker ./integration-tests/run.sh

You can also run integration tests on your (host) machine, which gives you some debugging ability, and typically runs faster:

  • ./integration-tests/run.sh

There are good debugging options for integration testing. See integration-tests/README.md.

Running unix commands in the container

  • scripts/run-in-docker bash

Accessing the local db

  • scripts/run-in-docker psql -d devdb

Config files

Config files (config/) define all env vars that you can use in Dark code. All config vars must start with DARK_CONFIG. Changing a config variable in config/dev requires restaring the devcontanier.

Debugging the client

You can enable the FluidDebugger by mousing over the Gear in the left-sidebar. There is also "Enable debugger" which enables a legacy debugger that nobody uses and doesn't work well.

If you're using Chrome, enable Custom Formatters to see ReScript values in Chrome Dev Tools instead of their JS representation. From within Chrome Dev Tools, click "⠇", "Settings", "Preferences", "Enable Custom Formatters".

Debugging dotnet

Debugger

The VSCode debugger works out of the box with Dark, supporting stepping, breakpoints, inspecting the stack, etc. You must launch the executable from VSCode for this to work--attaching does not currently seem to work. You can edit launch.json to change what tests are run or how other binaries are started up, which should be straightforward.

REPL (fsi)

You can get a REPL with all of the Dark libraries loaded by running:

Segfaults and crashes

When dotnet crashes, you can debug it by running:

  • lldb -- [your command]

In LLDB, you can use dotnet's SOS plugin to read the stack, values, etc. The plugin is automatically loaded in lldb in the dev container.

Production Services

The app is split into fsharp-backend and client/. Part of the backend is used in the client (Analysis).

These are compiled to create libraries and binaries.

These are put into containers, whose definitions are in containers/. We also have some containers which are defined entirely in their directory (typically, these have a self-contained codebase).

The containers are deployed via Kubernetes. A group of containers are deployed together, which is called a pod. Those pods, and how they are run (for example, how many of them, what secrets they have access to, how to check if they are still alive) are defined by a set of Yaml files which is called a deployment. Our deployments are all defined in the services directory.

A service in our repo typically wraps a deployment, but it can sometimes mean other things, so we also have a number of other services, defined via yaml files, in services. Some of the services are deployments that use 3rdparty containers (eg, "Let's Encrypt"), and some are abstractions around Google Cloud services.

Other important docs

Less important docs