# Search First Distributed Storage

## Project goal

This application is a containerized file storage system where users store files in object storage over a Web-UI, while searchable metadata is stored separately in Elasticsearch.

Separating file content from metadata enables fast filtering and search. Instead of scanning files directly, the system queries Elasticsearch, which is optimized for indexing and full-text search.

### Security

Security is enforced using JSON Web Token (JWT).

Workflow:

1. User authenticates.
2. Backend issues a signed JWT.
3. Each request includes the token.
4. Backend extracts the user ID from the token.
5. Elasticsearch queries and MinIO operations are scoped to that user.

This guarantees strict user isolation: each user can only access their own files and metadata.

### Redundancy and High Availability

The system is designed to avoid a single point of failure:

- MinIO: Two synchronized instances for file redundancy
- Elasticsearch: Two synchronized instances for metadata redundancy
- Backend (Node.js/Express): Two running instances behind Kubernetes

If one instance fails, the second continues serving requests to maintain availability.

A distributed file storage application with:

## Project Structure

- [`src/`](src/) - Angular frontend source
- [`server.js`](server.js) - Node.js backend (JWT auth, file APIs, Elasticsearch, MinIO)
- [`Dockerfile`](Dockerfile) - Multi-stage build for Angular + Node runtime
- [`docker-compose.yml`](docker-compose.yml) - Local full stack (app + Elasticsearch + MinIO)
- [`kubernetes/`](kubernetes/) - Kubernetes manifests

## Demo Login Users

- `alice` / `alice123`
- `bob` / `bob123`

Each user can only see files they uploaded (enforced with JWT + user-scoped metadata queries).

## Local Development (PowerShell)

NOTE: for load balancing use kuberentes deploy script

Install dependencies and start watch mode (frontend rebuild + backend restart):

```powershell
npm install --include=dev
npm run watch
```

App URL:

- `http://localhost:3000`

Notes:

- Backend expects Elasticsearch on `:9200` and MinIO on `:9000`.
- For the easiest local setup, use Docker Compose.

## Local Full Stack with Docker Compose (PowerShell)

Starts:

- App: `http://localhost:3000`
- Elasticsearch: `http://localhost:9200`
- MinIO API: `http://localhost:9000`
- MinIO Console: `http://localhost:9001`

```powershell
docker compose up --build
```

MinIO credentials:

- Username: `minioadmin`
- Password: `minioadmin`

## Docker Build and Run (PowerShell)

Build the application image:

```powershell
docker build -t search-first-dis-storage .
```

Run the container:

```powershell
docker run --rm -p 3000:3000 search-first-dis-storage
```

## Run on Kubernetes (Kind) (PowerShell)

Prerequisites:

- Docker Desktop running
- `kubectl` installed
- `kind` installed

One-command deployment:

```powershell
.\scripts\deploy-cluster.ps1
```

Optional (recreate cluster from scratch):

```powershell
.\scripts\deploy-cluster.ps1 -RecreateCluster
```

Optional (skip rebuild and reuse an existing local image tag):

```powershell
.\scripts\deploy-cluster.ps1 -SkipBuildApiImage
```

Create the local cluster (1 control-plane + 2 workers):

```powershell
kind create cluster --name dsys --config .\kubernetes\kind-config.yaml
```

Create required namespaces:

```powershell
kubectl create namespace search
kubectl create namespace storage
```

Deploy Elasticsearch + Service:

```powershell
kubectl apply -f .\kubernetes\elasticsearch.yaml
kubectl apply -f .\kubernetes\elasticsearch-service.yaml
```

Deploy MinIO:

```powershell
kubectl apply -f .\kubernetes\minio.yaml
```

Deploy API + HPA:

```powershell
kubectl apply -f .\kubernetes\node.yaml
kubectl apply -f .\kubernetes\api-hpa.yaml
```

Verify rollout:

```powershell
kubectl get pods -A
kubectl get hpa
```

Expose API locally (current manifests do not define an API Service, so port-forward the deployment):

```powershell
kubectl port-forward deployment/api 3000:3000
```

Then open:

- `http://localhost:3000`

Delete cluster when done:

```powershell
kind delete cluster --name dsys
```

## Push Image to Docker Hub (PowerShell)

Replace `<your-dockerhub-username>` with your Docker Hub username.

```powershell
$ImageName = "search-first-dis-storage"
$DockerHubUser = "piranhapedro"
$Tag = "latest"

docker login
docker build -t $ImageName .
docker tag $ImageName "${DockerHubUser}/${ImageName}:$Tag"
docker push "${DockerHubUser}/${ImageName}:$Tag"
```

## API Endpoints

- `POST /api/auth/login` - Returns JWT token
- `GET /api/health` - Returns health + dependency status
- `GET /health` - Alias to `/api/health`
- `POST /api/crash` - Intentionally triggers an unhandled exception to crash one backend instance
- `GET /api/files` - Lists current user's files (JWT required)
- `POST /api/files` - Uploads file + metadata (`file`, `filename`, `tags`) (JWT required)
- `GET /api/files/:id/download` - Downloads owned file (JWT required)

## k6 Upload + Crash Load Test

This test does both:

1. Upload all files in `testData/` concurrently, repeated multiple rounds.
2. Start a smaller upload phase and trigger `/api/crash` in the middle to kill one server instance.

Run:

```powershell
.\scripts\run-k6-upload-crash.ps1 `
  -BaseUrl "http://localhost:3000" `
  -BulkVUs 20 `
  -BulkRounds 3 `
  -SmallVUs 6 `
  -SmallIterationsPerVu 12 `
  -SmallFileCount 5 `
  -CrashDelaySeconds 10
```

Script files:

- [`scripts/run-k6-upload-crash.ps1`](scripts/run-k6-upload-crash.ps1)
- [`scripts/k6/upload-and-crash.js`](scripts/k6/upload-and-crash.js)

Run k6 against Kubernetes:

1. Keep `kubectl port-forward deployment/api 3000:3000` running in one terminal.
2. Run the same k6 script with `-BaseUrl "http://localhost:3000"` in another terminal.

## Argo CD (Kubernetes Cluster Management)

Deploy Kubernetes with Argo CD bootstrap (single command):

```powershell
.\scripts\deploy-kubernetes.ps1
```

If no cluster/context is configured yet and you want to auto-create a Kind cluster first:

```powershell
.\scripts\deploy-kubernetes.ps1 -CreateKindClusterIfMissing
```

When this option is used and `kind` is missing, the script attempts automatic installation (`winget`, then `choco` fallback).

Kind requires a running Docker daemon (Docker Desktop with Linux engine on Windows).

What this does:

- Installs Argo CD in namespace `argocd`
- Exposes Argo CD server on NodePort `30080`
- Creates/updates Argo CD Application `dsys-ct-lehmann`
- Uses git `origin` as `repoURL` and current git branch as `targetRevision`
- Triggers refresh so Argo starts reconciling right away

If your `origin` is private, add repository credentials in Argo CD first (SSH key or HTTPS token), otherwise sync will fail with a repository access error.

Low-level Argo install script (usually not needed directly):

```powershell
.\scripts\install-argocd.ps1
```

Get the initial admin password:

```powershell
.\scripts\argocd-admin-password.ps1
```

Access the UI:

- NodePort (if reachable in your Kind setup): `https://localhost:30080`
- Or port-forward:

```powershell
.\scripts\argocd-port-forward.ps1
```

Then open `https://localhost:8080` and log in with:

- Username: `admin`
- Password: output of `.\scripts\argocd-admin-password.ps1`
