Update README.md to reflect enhancements in POE Sensor - Modbus to MQTT Bridge Service. Expanded project description to include support for CO2 sensors and comprehensive monitoring features. Revised project structure and configuration details, including multi-sensor support and environment variable configuration. Enhanced documentation on MQTT topic structure, health check responses, and deployment options, ensuring clarity for users and contributors.
This commit is contained in:
Naab2k3
336
README.md
336
README.md
@ -1,18 +1,19 @@
|
|||||||
# Modbus to MQTT Bridge Service
|
# POE Sensor - Modbus to MQTT Bridge Service
|
||||||
|
|
||||||
A Python service that reads temperature and humidity data from a Modbus TCP server and publishes the data to an MQTT broker.
|
A robust Python service that reads temperature, humidity, and CO2 data from multiple Modbus TCP sensors and publishes the data to an MQTT broker with comprehensive monitoring and alerting capabilities.
|
||||||
|
|
||||||
## 📁 Project Structure
|
## 📁 Project Structure
|
||||||
|
|
||||||
```
|
```
|
||||||
POE/
|
POE-sensor/
|
||||||
├── config.py # Configuration constants
|
├── config.py # Configuration constants and sensor definitions
|
||||||
├── sensor_bridge.py # Core logic for Modbus and MQTT operations
|
├── sensor_bridge.py # Core logic for Modbus and MQTT operations
|
||||||
├── main.py # Entry point
|
├── sensor_tracker.py # Sensor status tracking and alerting system
|
||||||
|
├── health_check.py # HTTP health check server (optional monitoring)
|
||||||
|
├── main.py # Entry point and application lifecycle management
|
||||||
├── requirements.txt # Python dependencies
|
├── requirements.txt # Python dependencies
|
||||||
├── README.md # This file
|
├── poe-sensor.nomad # Nomad deployment configuration
|
||||||
├── poe.py # Original single file (backup)
|
└── README.md # This documentation
|
||||||
└── memory-bank/ # Memory bank directory
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🚀 Quick Start
|
## 🚀 Quick Start
|
||||||
@ -25,13 +26,17 @@ pip install -r requirements.txt
|
|||||||
### 2. Configure Settings
|
### 2. Configure Settings
|
||||||
Edit `config.py` to match your environment:
|
Edit `config.py` to match your environment:
|
||||||
```python
|
```python
|
||||||
# Modbus configuration
|
# Modbus sensor configuration
|
||||||
MODBUS_HOST = "10.84.48.153"
|
MODBUS_HOSTS = [
|
||||||
MODBUS_PORT = 505
|
{"ip": "10.84.60.31", "location": "Warehouse-B", "type": "temperature_humidity"},
|
||||||
|
{"ip": "10.84.60.37", "location": "Office", "type": "cwt_co2"},
|
||||||
|
# Add more sensors as needed
|
||||||
|
]
|
||||||
|
|
||||||
# MQTT configuration
|
# MQTT configuration
|
||||||
MQTT_BROKER = "mqtt.service.mesh"
|
MQTT_BROKER = "mqtt.service.mesh"
|
||||||
MQTT_TOPIC = "Temperature_Humidity"
|
MQTT_USERNAME = "relay"
|
||||||
|
MQTT_PASSWORD = "your_password"
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Run the Service
|
### 3. Run the Service
|
||||||
@ -41,130 +46,247 @@ python main.py
|
|||||||
|
|
||||||
## 📊 Features
|
## 📊 Features
|
||||||
|
|
||||||
- **Modbus TCP Client**: Reads data from Modbus holding registers
|
### Core Functionality
|
||||||
- **MQTT Publisher**: Publishes sensor data to MQTT broker
|
- **Multi-Sensor Support**: Handles both temperature/humidity and CO2 sensors
|
||||||
|
- **Modbus TCP Client**: Reads data from multiple Modbus holding registers
|
||||||
|
- **MQTT Publisher**: Publishes sensor data to structured MQTT topics
|
||||||
- **Data Processing**: Converts raw sensor values to calibrated readings
|
- **Data Processing**: Converts raw sensor values to calibrated readings
|
||||||
- **Error Handling**: Robust error handling and retry mechanisms
|
- **Error Handling**: Robust error handling and retry mechanisms
|
||||||
- **Logging**: Comprehensive logging for monitoring and debugging
|
- **Resource Management**: Automatic connection cleanup and garbage collection
|
||||||
- **Health Check API**: HTTP endpoints for system health monitoring
|
|
||||||
- **Sensor Status Tracking**: Real-time tracking of individual sensor health
|
|
||||||
- **Alerting System**: Automatic alerts for sensor failures and recovery
|
|
||||||
- **Recovery Detection**: Detects when failed sensors come back online
|
|
||||||
|
|
||||||
## 📈 Data Format
|
### Monitoring & Alerting
|
||||||
|
- **Real-time Sensor Tracking**: Individual sensor health monitoring
|
||||||
|
- **Automatic Alerting**: MQTT alerts for sensor failures and recovery
|
||||||
|
- **Health Check API**: Optional HTTP endpoints for system monitoring
|
||||||
|
- **Comprehensive Logging**: Detailed logging for debugging and monitoring
|
||||||
|
- **Status Reporting**: Real-time sensor status via MQTT topics
|
||||||
|
|
||||||
The service publishes JSON data to the MQTT topic:
|
### Production Ready
|
||||||
|
- **Nomad Deployment**: Ready-to-deploy Nomad job configuration
|
||||||
|
- **Docker Support**: Containerized deployment with dependency management
|
||||||
|
- **Environment Configuration**: Environment variable support for flexible deployment
|
||||||
|
- **Graceful Shutdown**: Proper cleanup and signal handling
|
||||||
|
|
||||||
|
## 📈 Data Format & Topics
|
||||||
|
|
||||||
|
### MQTT Topic Structure
|
||||||
|
The service uses a hierarchical topic structure for better organization:
|
||||||
|
|
||||||
|
```
|
||||||
|
Location/{location_name}/{sensor_type}/Time
|
||||||
|
Location/{location_name}/{sensor_type}/Status
|
||||||
|
Location/{location_name}/{sensor_type}/Data/temperature
|
||||||
|
Location/{location_name}/{sensor_type}/Data/humidity
|
||||||
|
Location/{location_name}/{sensor_type}/Data/co2 # CO2 sensors only
|
||||||
|
Location/{location_name}/{sensor_type}/alerts # Alert messages
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example Topics
|
||||||
|
```
|
||||||
|
Location/Warehouse-B/temperature-humidity/Time → "2024-01-15 10:30:25"
|
||||||
|
Location/Warehouse-B/temperature-humidity/Status → "online"
|
||||||
|
Location/Warehouse-B/temperature-humidity/Data/temperature → "23.5"
|
||||||
|
Location/Warehouse-B/temperature-humidity/Data/humidity → "65.2"
|
||||||
|
|
||||||
|
Location/Office/CO2-gas/Time → "2024-01-15 10:30:25"
|
||||||
|
Location/Office/CO2-gas/Status → "online"
|
||||||
|
Location/Office/CO2-gas/Data/temperature → "24.1"
|
||||||
|
Location/Office/CO2-gas/Data/humidity → "58.3"
|
||||||
|
Location/Office/CO2-gas/Data/co2 → "420"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Alert Format
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"time": "2024-01-15 10:30:25",
|
"alert_type": "sensor_failure",
|
||||||
"location": "Office",
|
"timestamp": "2024-01-15T10:30:25Z",
|
||||||
"temperature": 23.5,
|
"sensor_id": "10.84.60.31_Warehouse-B",
|
||||||
"humidity": 65.2
|
"sensor_ip": "10.84.60.31",
|
||||||
|
"sensor_location": "Warehouse-B",
|
||||||
|
"sensor_type": "temperature_humidity",
|
||||||
|
"consecutive_failures": 3,
|
||||||
|
"last_error": "Connection timeout",
|
||||||
|
"severity": "critical"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🔧 Configuration
|
## 🔧 Configuration
|
||||||
|
|
||||||
All configuration is centralized in `config.py`:
|
### Sensor Configuration
|
||||||
|
Add sensors to the `MODBUS_HOSTS` list in `config.py`:
|
||||||
|
|
||||||
| Parameter | Description | Default |
|
```python
|
||||||
|-----------|-------------|---------|
|
MODBUS_HOSTS = [
|
||||||
| `MODBUS_HOST` | Modbus TCP server IP | "10.84.48.153" |
|
# Temperature/Humidity sensors
|
||||||
| `MODBUS_PORT` | Modbus TCP port | 505 |
|
{"ip": "10.84.60.31", "location": "Warehouse-B", "type": "temperature_humidity"},
|
||||||
| `MQTT_BROKER` | MQTT broker address | "mqtt.service.mesh" |
|
{"ip": "10.84.60.32", "location": "Warehouse-C", "type": "temperature_humidity"},
|
||||||
| `MQTT_TOPIC` | MQTT publish topic | "Temperature_Humidity" |
|
|
||||||
| `PUBLISH_INTERVAL` | Data publish interval (seconds) | 10 |
|
|
||||||
| `LOCATION` | Sensor location identifier | "Office" |
|
|
||||||
|
|
||||||
## 📡 Sensor Mapping
|
# CO2 sensors (CWT series)
|
||||||
|
{"ip": "10.84.60.37", "location": "Office", "type": "cwt_co2"},
|
||||||
- **Register 0**: Temperature (raw value × 0.1 - 40)
|
]
|
||||||
- **Register 1**: Humidity (raw value × 0.1)
|
|
||||||
|
|
||||||
## 🏥 Health Check & Monitoring
|
|
||||||
|
|
||||||
### Health Check Endpoints
|
|
||||||
|
|
||||||
The service provides HTTP endpoints for monitoring:
|
|
||||||
|
|
||||||
- **http://localhost:8080/health** - Basic service health check
|
|
||||||
- **http://localhost:8080/sensors** - Detailed sensor status information
|
|
||||||
|
|
||||||
### Alerting System
|
|
||||||
|
|
||||||
The system automatically sends MQTT alerts for:
|
|
||||||
|
|
||||||
- **Sensor Failures**: When a sensor fails 3 consecutive times
|
|
||||||
- **Sensor Recovery**: When a failed sensor comes back online
|
|
||||||
|
|
||||||
Alert topics:
|
|
||||||
- `sensor-alerts` - Failure and recovery alerts
|
|
||||||
- `sensor-status/{location}/status` - Individual sensor status updates
|
|
||||||
|
|
||||||
### Demo Monitoring Script
|
|
||||||
|
|
||||||
Use the demo script to test monitoring features:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Check health endpoint
|
|
||||||
python demo_monitoring.py health
|
|
||||||
|
|
||||||
# Check sensors status
|
|
||||||
python demo_monitoring.py sensors
|
|
||||||
|
|
||||||
# Monitor system for 5 minutes
|
|
||||||
python demo_monitoring.py monitor 300
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Configuration Options
|
### Environment Variables
|
||||||
|
The service supports environment variable configuration:
|
||||||
|
|
||||||
| Parameter | Description | Default |
|
| Variable | Description | Default |
|
||||||
|-----------|-------------|---------|
|
|----------|-------------|---------|
|
||||||
| `HEALTH_CHECK_ENABLED` | Enable/disable health check server | True |
|
| `MQTT_BROKER` | MQTT broker address | "mqtt.service.mesh" |
|
||||||
| `HEALTH_CHECK_PORT` | HTTP server port | 8080 |
|
| `MQTT_PORT` | MQTT broker port | 1883 |
|
||||||
| `ALERTING_ENABLED` | Enable/disable MQTT alerts | True |
|
| `MQTT_USERNAME` | MQTT username | "relay" |
|
||||||
|
| `MQTT_PASSWORD` | MQTT password | - |
|
||||||
|
| `PUBLISH_INTERVAL` | Data publish interval (seconds) | 10 |
|
||||||
|
| `HEALTH_CHECK_ENABLED` | Enable HTTP health server | true |
|
||||||
|
| `HEALTH_CHECK_PORT` | Health server port | 8080 |
|
||||||
|
| `ALERTING_ENABLED` | Enable MQTT alerts | true |
|
||||||
| `SENSOR_TIMEOUT_THRESHOLD` | Failures before alert | 3 |
|
| `SENSOR_TIMEOUT_THRESHOLD` | Failures before alert | 3 |
|
||||||
| `RECOVERY_CONFIRMATION_COUNT` | Successes to confirm recovery | 2 |
|
| `RECOVERY_CONFIRMATION_COUNT` | Successes to confirm recovery | 2 |
|
||||||
|
|
||||||
## 🛠️ Maintenance
|
## 🏥 Health Check & Monitoring
|
||||||
|
|
||||||
### Running as a Service
|
### Optional Health Check Server
|
||||||
|
The service includes an optional HTTP health check server for monitoring:
|
||||||
|
|
||||||
For production deployment, consider running as a system service:
|
- **http://localhost:8080/health** - Basic service health status
|
||||||
|
- **http://localhost:8080/sensors** - Detailed sensor status information
|
||||||
|
|
||||||
```bash
|
### Health Response Example
|
||||||
# Example systemd service file
|
```json
|
||||||
[Unit]
|
{
|
||||||
Description=Modbus MQTT Bridge
|
"status": "healthy",
|
||||||
After=network.target
|
"timestamp": "2024-01-15T10:30:25Z",
|
||||||
|
"service": "poe-sensor",
|
||||||
[Service]
|
"version": "1.0.0",
|
||||||
Type=simple
|
"sensors_total": 7,
|
||||||
User=pi
|
"sensors_online": 6,
|
||||||
WorkingDirectory=/path/to/POE
|
"sensors_health": 85.7
|
||||||
ExecStart=/usr/bin/python3 main.py
|
}
|
||||||
Restart=always
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=multi-user.target
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Monitoring
|
### Sensor Status Response
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"timestamp": "2024-01-15T10:30:25Z",
|
||||||
|
"total_sensors": 7,
|
||||||
|
"online_sensors": 6,
|
||||||
|
"offline_sensors": 1,
|
||||||
|
"sensors": {
|
||||||
|
"10.84.60.31_Warehouse-B": {
|
||||||
|
"ip": "10.84.60.31",
|
||||||
|
"location": "Warehouse-B",
|
||||||
|
"type": "temperature_humidity",
|
||||||
|
"status": "online",
|
||||||
|
"last_success": "2024-01-15T10:29:45Z",
|
||||||
|
"uptime_percentage": 98.5
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
The service provides comprehensive logging. Monitor the logs for:
|
## 🚀 Production Deployment
|
||||||
- Connection status
|
|
||||||
- Data reading success/failure
|
### Nomad Deployment
|
||||||
- MQTT publish status
|
The service is designed for production deployment using HashiCorp Nomad:
|
||||||
- Error conditions
|
|
||||||
|
```bash
|
||||||
|
# Deploy to Nomad cluster
|
||||||
|
nomad job run poe-sensor.nomad
|
||||||
|
|
||||||
|
# Check deployment status
|
||||||
|
nomad job status poe-sensor
|
||||||
|
|
||||||
|
# View logs
|
||||||
|
nomad alloc logs <allocation_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key Deployment Features
|
||||||
|
- **No Health Check Dependencies**: Service is considered healthy when running
|
||||||
|
- **Fast Deployment**: Minimal startup time requirements
|
||||||
|
- **Auto-restart**: Automatic restart on application crashes
|
||||||
|
- **Resource Limits**: Configured CPU and memory limits
|
||||||
|
- **Log Management**: Automatic log rotation and retention
|
||||||
|
|
||||||
|
### Docker Configuration
|
||||||
|
The Nomad job uses a pre-configured Docker image with:
|
||||||
|
- Python 3.x runtime
|
||||||
|
- Required system dependencies
|
||||||
|
- Automatic dependency installation
|
||||||
|
- Git-based source code deployment
|
||||||
|
|
||||||
|
## 📡 Sensor Support
|
||||||
|
|
||||||
|
### Temperature/Humidity Sensors
|
||||||
|
- **Register 0**: Temperature (formula: (125 - (-40)) * raw / 1650 - 40)
|
||||||
|
- **Register 1**: Humidity (formula: raw * 100 / 1000)
|
||||||
|
- **Output**: Temperature in °C, Humidity in %RH
|
||||||
|
|
||||||
|
### CWT CO2 Sensors
|
||||||
|
- **Register 0**: Humidity (0.1%RH resolution)
|
||||||
|
- **Register 1**: Temperature (0.1°C resolution, 2's complement)
|
||||||
|
- **Register 2**: CO2 (1ppm resolution)
|
||||||
|
- **Output**: Temperature in °C, Humidity in %RH, CO2 in ppm
|
||||||
|
|
||||||
|
## 🔧 Troubleshooting
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
1. **Sensor Connection Failures**
|
||||||
|
- Check network connectivity to sensor IPs
|
||||||
|
- Verify Modbus port (default: 505) is accessible
|
||||||
|
- Check sensor power and network configuration
|
||||||
|
|
||||||
|
2. **MQTT Connection Issues**
|
||||||
|
- Verify MQTT broker address and credentials
|
||||||
|
- Check network connectivity to MQTT broker
|
||||||
|
- Review MQTT broker logs for connection attempts
|
||||||
|
|
||||||
|
3. **Health Check Not Responding**
|
||||||
|
- Health check server is optional and non-blocking
|
||||||
|
- Service will continue running even if health server fails
|
||||||
|
- Check port 8080 availability if health check is needed
|
||||||
|
|
||||||
|
### Log Analysis
|
||||||
|
Monitor logs for key indicators:
|
||||||
|
- **Connection status**: Modbus and MQTT connection messages
|
||||||
|
- **Data reading**: Successful sensor data retrieval
|
||||||
|
- **Publishing status**: MQTT message publication confirmation
|
||||||
|
- **Error conditions**: Connection failures and retry attempts
|
||||||
|
- **Sensor health**: Failure/recovery notifications
|
||||||
|
|
||||||
|
### Performance Monitoring
|
||||||
|
- **Cycle completion**: Monitor sensor processing cycles
|
||||||
|
- **Memory usage**: Automatic garbage collection logs
|
||||||
|
- **Alert frequency**: Sensor failure and recovery alerts
|
||||||
|
- **Uptime percentage**: Individual sensor reliability metrics
|
||||||
|
|
||||||
## 🤝 Contributing
|
## 🤝 Contributing
|
||||||
|
|
||||||
1. Fork the repository
|
1. Fork the repository
|
||||||
2. Create a feature branch
|
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
|
||||||
3. Make your changes
|
3. Make your changes with proper testing
|
||||||
4. Test thoroughly
|
4. Commit your changes (`git commit -m 'Add amazing feature'`)
|
||||||
5. Submit a pull request
|
5. Push to the branch (`git push origin feature/amazing-feature`)
|
||||||
|
6. Open a Pull Request
|
||||||
|
|
||||||
|
### Development Guidelines
|
||||||
|
- Follow Python PEP 8 style guidelines
|
||||||
|
- Add comprehensive logging for new features
|
||||||
|
- Test with multiple sensor types
|
||||||
|
- Update documentation for configuration changes
|
||||||
|
- Ensure backward compatibility when possible
|
||||||
|
|
||||||
## 📝 License
|
## 📝 License
|
||||||
|
|
||||||
This project is licensed under the MIT License.
|
This project is licensed under the MIT License - see the LICENSE file for details.
|
||||||
|
|
||||||
|
## 🆘 Support
|
||||||
|
|
||||||
|
For issues and questions:
|
||||||
|
1. Check the troubleshooting section above
|
||||||
|
2. Review application logs for error details
|
||||||
|
3. Verify network connectivity and sensor configuration
|
||||||
|
4. Create an issue in the project repository with:
|
||||||
|
- Detailed error description
|
||||||
|
- Relevant log excerpts
|
||||||
|
- Environment configuration
|
||||||
|
- Steps to reproduce the issue
|
||||||
Reference in New Issue
Block a user