AWS Lambda-based health data ingestion pipeline for UltraHuman devices with JWT-secured reporting, automated participant discovery, and S3 parquet storage
Expert guidance for developing and deploying the SensorFabric UltraHuman (UH) AWS Lambda-based data pipeline that collects health/fitness metrics from UltraHuman devices.
This is a multi-Lambda serverless pipeline with four core functions:
1. **biobayb_uh_publisher** - Scheduled daily participant discovery and SNS message publishing
2. **biobayb_uh_uploader** - SNS-triggered data collection from UltraHuman API to S3 parquet files
3. **biobayb_uh_template_generator** - JWT-authenticated weekly health report generation
4. **biobayb_uh_jwt_generator** - Weekly JWT token generation for secure report access
Scheduled Trigger → Participant Discovery → SNS Publish → API Data Collection → Parquet Storage → MDH Update → Weekly JWT Generation
**Full deployment pipeline (recommended):**
```bash
./deploy.sh
```
**Deploy with CDK:**
```bash
./deploy.sh --cdk
```
**Deploy specific stack only:**
```bash
./deploy.sh --stack uh-biobayb-dev
```
**Build container images without deployment:**
```bash
./deploy.sh --build-only
```
**Deploy only (assumes ECR images exist):**
```bash
./deploy.sh --deploy-only
```
**Run health checks on deployed functions:**
```bash
./deploy.sh --health-check
```
**Test deployed functions:**
```bash
./deploy.sh --test
```
**Rollback to previous deployment:**
```bash
./deploy.sh --rollback
```
Navigate to CDK directory and set up:
```bash
cd cdk/
pip install -r requirements.txt
```
**Bootstrap CDK (first deployment only):**
```bash
cdk bootstrap
```
**Deploy infrastructure:**
```bash
cdk deploy --all
```
**Preview changes before deployment:**
```bash
cdk diff
```
**Destroy infrastructure:**
```bash
cdk destroy
```
The system supports two deployment environments with isolated resources:
All functions share the same Docker image from ECR with different CMD handlers:
S3 parquet files are organized by participant and date:
```
s3://uoa-biobayb-uh-{env}/
├── participant_id=xxx/
│ ├── date=2024-01-01/
│ │ └── data.parquet
│ └── date=2024-01-02/
│ └── data.parquet
```
The template generator implements JWT-based authentication:
Send EventBridge event:
```json
{
"source": ["sensorfabric.manual"],
"detail-type": ["UltraHuman Data Upload Request"]
}
```
Send EventBridge event:
```json
{
"source": ["sensorfabric.manual"],
"detail-type": ["UltraHuman JWT Generation Request"],
"detail": {
"start_date": "2024-01-01",
"end_date": "2024-01-31",
"participant_id": "optional_specific_participant"
}
}
```
**View real-time function logs:**
```bash
aws logs tail /aws/lambda/{function_name} --follow
```
**Test function invocation:**
```bash
aws lambda invoke --function-name {function_name} /tmp/response.json
```
**Check CloudFormation deployment events:**
```bash
aws cloudformation describe-stack-events --stack-name {stack_name}
```
**Monitor failed executions:**
Check SQS DLQ for messages after 2 retry attempts
Before working with this project, ensure:
Leave a review
No reviews yet. Be the first to review this skill!
# Download SKILL.md from killerskills.ai/api/skills/sensorfabric-ultrahuman-lambda-pipeline/raw