If you are interested in other Enphase information the following other pages may also be of interest:
What is inside the Enphase Envoy-S (teardown)
Reverse Engineering the Enphase Installer Toolkit
Enphase Envoy-S Open Ports!
I’ve recently had to interface with an EnPhase Envoy Solar PV system. The annoying thing is the lack of documentation.
The API for the “cloud service” exists and is well documented but the API for the local device itself doesn’t seem to exist.
There is one tiny one page document that seems to suggest you can get some data from the device. The information gleaned from their example API ” http://device.ip.address.here/api/v1/production ” is poor.
{
"wattHoursToday": 6641,
"wattHoursSevenDays": 6641,
"wattHoursLifetime": 6669,
"wattsNow": 90
}
First.. it gives far LESS information than you can just retrieve from the main web page for the device on the LAN. Secondly the wattsNow field doesn’t seem to match the information displayed on the main page of the device. The wattsHoursSevenDays is also slightly off. Third.. it doesn’t even update regularly. You can’t have a system that detects dips in production due to cloud cover etc.
However. I did find some other sensible sources for information – and information that updates at an even more regular rate than the devices main website!
When you visit the site on the device running firmware D4.2.27 it makes the following requests:
http://10.0.0.177/backbone/application.js?version=04.02.43
This URL contains a lot of references to other json files that I need to research. See further down in this post for the other URLs exposed.
http://10.0.0.177/home.json
This json file contains the Database size / utilisation, date and time on the device, connection status and settings and update status.
http://10.0.0.177/production.json
Contains most useful information.. Production and (if fitted) consumption data!
{
"production":[
{
"type":"inverters",
"wNow":74,
"whLifetime":6815.014722222222,
"readingTime":1470250044,
"activeCount":14
},
{
"type":"eim",
"activeCount":1,
"whLifetime":6704.458,
"whLastSevenDays":6676.458,
"whToday":6676.458,
"wNow":66.236,
"rmsCurrent":1.179,
"rmsVoltage":246.433,
"reactPwr":276.774,
"apprntPwr":290.494,
"pwrFactor":0.23,
"readingTime":1470250044
}
],
"consumption":[
{
"type":"eim",
"activeCount":1,
"whLifetime":8025.821,
"whLastSevenDays":7899.821,
"whToday":7899.821,
"wNow":2641.386,
"varhLeadToday":2822.701,
"varhLagToday":1810.03,
"vahToday":9018.68,
"varhLeadLifetime":2951.701,
"varhLagLifetime":1922.03,
"vahLifetime":9473.68,
"rmsCurrent":11.924,
"rmsVoltage":246.458,
"reactPwr":-284.009,
"apprntPwr":2938.817,
"pwrFactor":0.9,
"readingTime":1470250044
}
]
}
http://10.0.0.177/inventory.json
Also contains some very useful information. Detailed status about the state of each micro-inverter. So far I’ve seen the following states:
– “envoy.global.ok”
– “envoy.cond_flags.pcu_ctrl.commandedreset”
– “envoy.cond_flags.pcu_ctrl.dc-pwr-low”
– “envoy.cond_flags.obs_strs.discovering”
– “envoy.cond_flags.obs_strs.failure”
The full conversion table on what these status messages mean in english is at the end of this article.
The full output of the page is as follows:
[
{
"type":"PCU",
"devices":[
{
"part_num":"800-00356-r04",
"installed":"1470228526",
"serial_num":"REDACTED",
"device_status":[
"envoy.cond_flags.pcu_ctrl.commandedreset",
"envoy.cond_flags.pcu_ctrl.dc-pwr-low"
],
"last_rpt_date":"1470249596",
"admin_state":1,
"created_date":"1470228526",
"img_load_date":"1424997903",
"img_pnum_running":"520-00045-r01-v01.22.00",
"ptpn":"540-00087-r01-v01.22.00",
"producing":true,
"communicating":true,
"chaneid":1627392273,
"device_control":[
{
"gficlearset":false
}
]
},
{
"part_num":"800-00356-r04",
"installed":"1470228529",
"serial_num":"REDACTED",
"device_status":[
"envoy.cond_flags.obs_strs.discovering"
],
"last_rpt_date":"1470249602",
"admin_state":1,
"created_date":"1470228529",
"img_load_date":"1424997903",
"img_pnum_running":"520-00045-r01-v01.22.00",
"ptpn":"540-00087-r01-v01.22.00",
"producing":false,
"communicating":true,
"chaneid":1627392529,
"device_control":[
{
"gficlearset":false
}
]
},
{
"part_num":"800-00356-r04",
"installed":"1470228533",
"serial_num":"REDACTED",
"device_status":[
"envoy.global.ok"
],
"last_rpt_date":"1470249613",
"admin_state":1,
"created_date":"1470228533",
"img_load_date":"1424997903",
"img_pnum_running":"520-00045-r01-v01.22.00",
"ptpn":"540-00087-r01-v01.22.00",
"producing":true,
"communicating":true,
"chaneid":1627392785,
"device_control":[
{
"gficlearset":false
}
]
}
]
}
]
And thanks to Stuart Graves:
/stream/meter
Which produces a continual stream of data for each phase. It does require authentication. It might be useful to read Stuart’s comment here.
All of this means you can make your own system that (in my case) refreshes generation and consumption status up-to-the-second!
A couple of notes: The Consumption metering / data seems to give strange low or even minus values if you fetch the json file more than roughly once per second. (Possibly because it works on a counter and the math goes wrong?)
During nighttime the Production metering goes into a minus value, for example -7 Watts. This is in fact the correct reading. The Envoy-S javascript filters out minus values and ensures a 0 is displayed. Cheeky!
The panel count on the main production.json file is the count of associated panels and not the currently generating panels. I’ve used the inventory.json file to count the number of panels “producing”, I refresh this information every 3 minutes instead of every second.
My next steps are to investigate the JavaScript and see what other pages are accessible. I also need to see if I can reverse engineer, find out or ask the installer for the “Installer Login” password (UPDATE: This has been done, see here!).
I also wish there was public documentation. No google searches for the status strings or other data I’ve found in the JSON files have returned any results to do with Enphase / SolarPV.
The backbone/application.js file exposes the following sub-files on the devices web server.
– /admin/lib/admin_dcc_display.json
– /ivp/tpm/capability
– /datatab/event_dt.rb?start=0&length=153
– /info.xml
– /ivp/meters
– /installer/pcu_comm_check
– /api/v1/production/inverters
– /prov
– /ivp/peb/newscan
– /installer/profiles/index.json
– /installer/profiles/details.json
– /installer/profiles/inverters_status.json
– /installer/profiles/set_profile.json
– /admin/lib/admin_pmu_display.json
– /ivp/peb/reportsettings
– /ivp/tpm/select
– /ivp/tpm/tpmstatus
– /ivp/tpm/parameters
– /admin/lib/dba.json
– /admin/lib/security_display.json
– /admin/lib/date_time_display.json?tzlist=1&locale=en
– /admin/lib/network_display.json
– /api/v1/production/inverters
– /admin/lib/wireless_display.json?site_info=0
Full status messages system strings to plain english:
cond_flags: {
acb_ctrl: {
bmuhardwareerror: "BMU Hardware Error",
bmuimageerror: "BMU Image Error",
bmumaxcurrentwarning: "BMU Max Current Warning",
bmusenseerror: "BMU Sense Error",
cellmaxtemperror: "Cell Max Temperature Error",
cellmaxtempwarning: "Cell Max Temperature Warning",
cellmaxvoltageerror: "Cell Max Voltage Error",
cellmaxvoltagewarning: "Cell Max Voltage Warning",
cellmintemperror: "Cell Min Temperature Error",
cellmintempwarning: "Cell Min Temperature Warning",
cellminvoltageerror: "Cell Min Voltage Error",
cellminvoltagewarning: "Cell Min Voltage Warning",
cibcanerror: "CIB CAN Error",
cibimageerror: "CIB Image Error",
cibspierror: "CIB SPI Error"
},
obs_strs: {
discovering: "Discovering",
failure: "Failure to report",
flasherror: "Flash Error",
notmonitored: "Not Monitored",
ok: "Normal",
plmerror: "PLM Error",
secmodeenterfailure: "Secure mode enter failure",
secmodeexitfailure: "Secure mode exit failure",
sleeping: "Sleeping"
},
pcu_chan: {
acMonitorError: "AC Monitor Error",
acfrequencyhigh: "AC Frequency High",
acfrequencylow: "AC Frequency Low",
acfrequencyoor: "AC Frequency Out Of Range",
acvoltage_avg_hi: "AC Voltage Average High",
acvoltagehigh: "AC Voltage High",
acvoltagelow: "AC Voltage Low",
acvoltageoor: "AC Voltage Out Of Range",
acvoltageoosp1: "AC Voltage Out Of Range - Phase 1",
acvoltageoosp2: "AC Voltage Out Of Range - Phase 2",
acvoltageoosp3: "AC Voltage Out Of Range - Phase 3",
agfpowerlimiting: "AGF Power Limiting",
dcresistancelow: "DC Resistance Low",
dcresistancelowpoweroff: "DC Resistance Low - Power Off",
dcvoltagetoohigh: "DC Voltage Too High",
dcvoltagetoolow: "DC Voltage Too Low",
dfdt: "AC Frequency Changing too Fast",
gfitripped: "GFI Tripped",
gridgone: "Grid Gone",
gridinstability: "Grid Instability",
gridoffsethi: "Grid Offset Hi",
gridoffsetlow: "Grid Offset Low",
hardwareError: "Hardware Error",
hardwareWarning: "Hardware Warning",
highskiprate: "High Skip Rate",
invalidinterval: "Invalid Interval",
pwrgenoffbycmd: "Power generation off by command",
skippedcycles: "Skipped Cycles",
vreferror: "Voltage Ref Error"
},
pcu_ctrl: {
alertactive: "Alert Active",
altpwrgenmode: "Alternate Power Generation Mode",
altvfsettings: "Alternate Voltage and Frequency Settings",
badflashimage: "Bad Flash Image",
bricked: "No Grid Profile",
commandedreset: "Commanded Reset",
criticaltemperature: "Critical Temperature",
"dc-pwr-low": "DC Power Too Low",
iuplinkproblem: "IUP Link Problem",
manutestmode: "In Manu Test Mode",
nsync: "Grid Perturbation Unsynchronized",
overtemperature: "Over Temperature",
poweronreset: "Power On Reset",
pwrgenoffbycmd: "Power generation off by command",
runningonac: "Running on AC",
tpmtest: "Transient Grid Profile",
unexpectedreset: "Unexpected Reset",
watchdogreset: "Watchdog Reset"
},
rgm_chan: {
check_meter: "Meter Error",
power_quality: "Poor Power Quality"
}
}
Some more info about errors can be found on page 45 of the manual.
CommandedReset Recommended Action: No action is required; it will automatically resume normal operation momentarily. Description: The microinverter was reset, either following a successful software download or by user command.
DcPowerTooLow Recommended Action: This condition should correct itself. No action is required. Description: This condition may occur at sunrise or sunset, while the modules are covered with snow, or during extreme weather. This event indicates that sunlight levels are too low for effective production. Once sunlight levels increase, the microinverter resumes power production and this event message will clear.
DcVoltageTooLow Recommended Action: This is usually a normal condition during hours of low light and at dawn and dusk. No action is required. Description: The microinverter reports that DC input voltage from the PV module is too low. If this condition does not clear during hours of full day
PowerOnReset Description: The microinverter has powered on after having DC disconnected.
thecomputerperson 
Pingback: Enphase Envoy-S Open Ports! | thecomputerperson
Bought the envoy-s 3 weeks ago and having the same difficulties in reading the local data.
Found several scripts for running man-in-the-middle option or scripts for OpenHAB, but haven’t found the information you wrote.
Unfortunately I have not that much information in the production.json, as you demonstrate.
I bookmark this link and follow your progress.
Thank you for commenting. What firmware version is your Envoy-s running ?
Does it show that you have micro-inverters connected and “producing” on the envoy-s web interface?
@license Backbone-Envoy 4.2.43 (2016-05-20)
Although the Enphase site clearly demonstrates the exact number of producing inverters, the only output I get from production.json is:
{“production”:[{“type”:”inverters”,”wNow”:46,”whLifetime”:477193.87611111114,”readingTime”:1470855222,”activeCount”:20}]}
Is that all you see in the production.json? In my case there is a “type”:”eim”, section and also a “consumption” section.
You should also see if /inventory.json contains anything interesting too.
I’ve not yet managed to work out how to get individual inverter generation statistics.
This evening my Envoy-S seems to have crashed or hung! This will be interesting to try and work out how to solve. It is responding to pings but the web interface is not returning any data.
Update: In the end I just had to turn off the power to the Envoy-S and turn it back on again.
Do you have an Envoy or an Envoy-S? The one I have is running firmware “D4.2.27 (b13066)” with a “Software Build Date” of “Jun 03, 2016 1:17 AM” which is strange. Your comment seems to show you have a _newer_ firmware but with a build date after the build date of my older firmware which doesn’t make much sense.
Also the request to the javascript file that my firmware makes references 04.02.43.. even more weird.
It’s an Envoy-S (EU version). The inventory page contains output which is more or less identical. Unfortunately the production.json not.
Not sure what to suggest :(
Thanks for the head start.
I have > D4.4.x
when prompted for username:password are envoy:nnnnnn where nnnnnn=last 6 digits of serial number
This works for wifi and diagnostics from home but not other screens :(
try
/home?locale=en&classic=1
/home?locale=en&classic=2
/production.json?details=1
/production.json?details=2 etc (might be interval records?)
to do
The installer toolkit can see all inverter details without a password – but how? Time to sniff some traffic.
Thanks for the info. I’ve yet to try the installer toolkit and prod the buttons on the Envoy. Will it work while the Envoy is on the normal network? (My one is hard wired / ethernet).
Yes and shows live inverter level data. It can connect to the envoy wirelrss ap but also works if device is on the same subnet. The app queries the presence of the envoy and it responds with its serial number which appears in the app. Touch this and your in.
You can see this arp traffic using wireshark but not http from app ??? May be limitation of windows on promiscuous mode can see http access from browser np..
I will give it a go and post an update. It will be at least a few weeks before I get a chance though. Too many things to do and not enough time.
also try
/info
/ivp/meters
Authentication of installer app process
https://s3.amazonaws.com:443/mobile-versions.enphaseenergy.com/app_versions.json
/info
app uses api key (based on info? eg S/N) and pings. This was different to the web API key!
xxapi/xamarin.com/api/ping?apikey=XXXXXXXXXXXXXXXXXXXXXXXX
there is some traffic back and forth inc some binary
x-pong
/api/collected?apikey= XXXXXXXXXXX¤tTime=xxxxxxx
/prov returns unauthorised
/prov again gets a response. It contains authorisation below
GET /prov HTTP/1.1
Accept: application/json
Authorization: Digest username=”installer”, realm=”enphaseenergy.com”, nonce=”xxxxxxxxxxxxxxxx”, uri=”http://192.168.0.x/prov”, response=”xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”, qop=auth, nc=00000001, cnonce=”xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”
Host: 192.168.0.x
Connection: Keep-Alive
Accept-Encoding: gzip
User-Agent: okhttp/2.7.
hmmmmm……. digest access auth
The information you are seeking about the energy each inverter is generating appears to be returned at this URL:
/api/v1/production/inverters
It requires authentication. In my case I was able to replay an authentication request that I had sniffed 13 hours earlier! So if you can sniff a successful authentication header from the installer toolkit request _to that specific page_ you may just be able to replay the authentication line from the header for eternity!
[{ "serialNumber": "REDACTED1", "lastReportDate": 1472215582, "lastReportWatts": 221, "maxReportWatts": 244 }, { "serialNumber": "REDACTED2", "lastReportDate": 1472215588, "lastReportWatts": 226, "maxReportWatts ": 248 }]While playing about I did manage to get /home.json to spit out:
"alerts": [{ "msg_key": "lwui.home.warnings.cross_domain_traffic" }],Interesting.
Unfortunately the amazon link of Paul isn’t working at the moment.
I also tried to get access through the installer toolkit, but with the lack of passwords, it wasn’t successful.
/api/v1/production/inverters requires name and password. I get exact the same data of my 20 inverters by filling in envoy and the last 6 numbers of the serial number.
I’ve fixed paul’s link – it was missing the https on the http:// bit.
Installer toolkit just let me right on on my system without the need to authenticate. Yet if I visit the same URLs on the browser it requires authentication.
As Paul noted, the app generates the authentication details somehow.
If you can sniff the details it seems the final authentication http header can be reused for eternity. It seems the envoy doesn’t check for expiry of the nonce.
Sadly whatever details are generated are specific to your unit so I can’t share my http request with you and have you authenticate against your Envoy with it :(
Ypu shouldnt need user .. pass with the installer toolkit.
It finds the envoy on the local lan or you can use the envoy ap lan.
The s/n will appear when its found. Just tap on this.
The Installer Toolkit application is a Xamarin (Mono) .net cross platform app. This means, when extracted, you get a load of (as far as I’m aware) windows compatible .dll files.
The authentication is likely handled by “assemblies\Enphase.InstallerToolkit.dll”.
Next step – I’ve got to learn how to reverse engineer .net dll files :< This might be beyond my abilities.
Pingback: Reverse Engineering the Enphase Installer Toolkit | thecomputerperson
Succeeded in getting access through the installer kit.
Tried to sniff the procedure with wireshark, which produced some files with http 1.1 or application/json information in it.
Tried to search for keys/passwords, but my lack of knowledge of wireshark makes it a slow process. I’ll keep on trying with your suggestions.
Anyone know how to pass the username/password to the envoy-s api via http post? I’m looking to grab the panel level info, and I can do so via:
http://192.168.1.10/api/v1/production/inverters
(192.168.1.10 is the wifi IP of my envoy-s)
But it needs a username of ‘envoy’ and password of the last 6 digits of the serial number, which it pops up and requests. I’ve tried:
http://192.168.1.10/api/v1/production/inverters&user=envoy&password=123456
(where 123456 is the last 6 digits of my envoy-s)
which returns:
{
“status”: 404,
“error”: “”,
“info”: “Resource /api/v1/production/inverters&user=envoy&password=123456 not found”,
“moreInfo”: “”
}
I’ve also tried the same thing but instead of & I used ?, and it popped up with the password prompt as if I wasn’t using them.
Anyone figured this one out yet?
What code are you using to scrape the data? Normally you just pass it extra parameters such as..
xmlhttp.URL(“http://192.168.1.10/api/v1/production/inverters”)
xmlhttp.Credentials(“envoy”,”034222″)
xmlhttp.Go()
(Very rough example, not working code!).
I was using a post–just pasting into the web page.
I believe that using a format similar to an ftp site works. For instance:
http://envoy:123456@192.168.1.10/api/v1/production/inverters
I need to test it on a separate computer.
Once I have that working, I’ll figure out what language I want to learn to do this…right now I know perl, but I think I should probably learn python.
Thanks for the reply…good to know there is a way to do it, and that I’m not barking up a dead tree!
I’d approach it the other way around. Mash out the code and get it to authenticate and dump the resulting page.
I’ve not got any experience in Perl but as far as I can see LWP takes authentication options:
use LWP 5.64; my $browser = LWP::UserAgent->new; $browser->credentials( '10.0.0.177:80', 'realmnamehere', 'usernamehere' => 'passwordhere' ); my $response = $browser->get($url);Perl looks pretty horrible from the code I’ve just seen, I expect Python is a far better bet :)
I’ve been working to programmatically authenticate the inverter production page via Matlab. The built in authentication options don’t work and the method suggested above of using http://user:pass@xxx doesn’t work either.
Do you have an Envoy or Envoy-S? My Envoy-S doesn’t need authentication for /inventory.json or /production.json so I can’t test against my own system and help you with code :(
I have an Envoy-S. I too don’t need authentication for /production but the real data I want is in /production/inverters. That is password protected.
Ah yes, that one is password protected but I’ve not yet figured out how to calculate the password from the envoy serial number. The older Envoy firmware was “installer” and the last 6 digits of the serial number (as far as I remember) but the newer firmware or Envoy-S seems to have a dynamically generated password.
For more information on my password generation probing see:
I must have an older unit then because user: envoy pass: XXXXXX (last 6 of serial) works for me.
Doesn’t work for me :( I am on Software Version D4.2.27 (b13066)
I was able to get to this page by using the ‘envoy’ username and last 6 digits of the serial number. Try:
http://envoy:123456@192.168.1.10/api/v1/production/inverters
(where you substitute the last 6 of your serial and your IP number)
The only odd thing is that it pops up with a prompt that you have to click yes to, not sure how that’ll work out programatically.
I haven’t had time to look into this, since I posted last.
That one works for me with the envoy/XXXXXX details!
{ "serialNumber": "REMOVED", "lastReportDate": 1475688141, "lastReportWatts": 0, "maxReportWatts": 259 },Matlab returns 401 on Dave’s link for me. Glad you that login info worked for you. I’m running software version R4.2.33 (b68db1)
I can download Matlab and test here. I’ve never used it before – what are the lines of code you have that make the request ?
It all revolves around the built in function “webread”. Documentation is here: https://www.mathworks.com/help/matlab/ref/webread.html Truthfully, Matlab is probably not the best language for this. It’s just the one they taught me in school. It’s also expensive.
My system has build D4.2.27 (b13066), for reference
It interesting that we have what appears to be very different firmware builds. I have the Envoy-S Metered (ENV-S-AM1-120). I would guess you both do not have the metered version.
I have a consumption and production clamp meter that came with the Envoy-S and the data read out shows on the Envoy page.
It looks like matlab is just sending “Basic” authentication in a single request and not triggering a Digest authentication (First a non-authenticated request needs to be made in which the envoy provides a salt / nonce which matlab [or other software] then hashes the authentication details with to produce a unique authentication string before making the second authenticated request with this string).
The following might help but I can’t actually see any code in that example about authentication even though it claims it tackles Digest auth.
https://uk.mathworks.com/help/matlab/matlab_external/send-http-message.html
Wireshark is a great tool – Install it and use the filter “host 10.0.0.177” where 10.0.0.177 is the IP of your envoy. Make a request / run your matlab script and you can then right click on a request and go to Follow –> TCP Stream to see if matlab sent a Basic or Digest authentication line.
Just had a breakthrough. First, I took my own advice and got away from Matlab. I grabbed Visual Studio and started writing a console app in C#. The following code grabs the json file:
var envoy = new WebClient { Credentials = new NetworkCredential(user, pw) };
var response = envoy.DownloadString(url);
It was only a matter of time before someone blogged about this stuff! I currently have OpenHAB polling my envoy-s production.json and logging production and consumption data, and also activating heating etc based on available power! :) The APIs they provide are USELESS….!
I’ve also had my envoy-s metered crash twice now since I began polling it every few seconds or so. It required a hard reset. I have no evidence of course that it wouldn’t have crashed anyway, but it is annoying. Has anyone else noticed this, or has anyone had their envoy-s crash when not polling it? (you get an email from enlighten saying it is not reporting)
I’ve certainly had it crash twice. Both times I thought it was due to my polling and it seemed to crash in an extended amount of time when I changed the polling to 2 seconds instead of 1 second etc.. but I don’t recall changing my polling code since and I’ve not had it crash (yet).
If you poll the Envoy consumption data more than once per second you get totally screwed up values (minus values and the 7 day consumption total starts counting down!).
Having said that – it crashed again today. Lots of other random services still respond but port 80 doesn’t ever load and the Envoy stops reporting to the cloud service. Boring. Wish I could reboot it remotely :<
Jam: I’ve been testing. I thought maybe the crash was triggered by either.
1) Requests being sent too quickly to the Envoy
2) A hard-limit on the number of requests that the envoy can handle before it runs out of ram, temp filesystem space or something
3) Race condition
The number of requests I seemed to send before it crashed was around 604,800 however.. I’ve been soak testing it with a script that has been hammering the Envoy with over 100 requests per minute and I’ve reached well over 800,000 requests and it still hasn’t crashed.
There goes my theory.
Which files are you fetching every second? I was testing with /production.json
I wish I had SSH or some other debug access to the device to either restart it remotely or, even better, figure out what is causing the lock up.
Hi tcp, I really appreciate you hammering your system in the name of science!
I am only fetching /production.json, once every 10 seconds on a little wifi display I built to have in the kitchen, and once every 30s on OpenHAB.
I really think getting the installer password used by the installer app could possibly open up some options for getting further access to the box.
I’m surprised they don’t have some kind of watchdog running on it to restart the service – it does stop it uploading to enlighten, and does stop logging data when it “crashes” so it’s kind of important….
Some good info here, thanks. I’m using Splunk to fetch and display the data from my Envoy-S … there’s a short forum post here: https://answers.splunk.com/answers/390296/how-to-get-splunk-to-fetch-json-data-from-enphase.html#comment-451860
Over the last week my Envoy updated itself from “Software Version D4.2.27 (b13066)” to “Software Version D4.4.89 (5e7c1d)”.
I now find that the web interface (html and everything) takes many minutes to load up and the json files take a similar amount of time. Sometimes it takes 1.4 minutes, sometimes it takes 3.5 minutes, sometimes it takes 5 minutes to just return one of the json pages.
The web interface now mentions batteries which I believe have recently been launched as an Enphase product in several countries.
Anyone else had this firmware? Anyone else having page load delay problems?
Same firmware has upgraded itself here – no delay problem. Also haven’t had any crashes for ages!
Thanks.. I will monitor and see what is going on. Seems to be ok for a bit and then goes weird, possibly when I request inventory.json and production.json in very quick succession.
Right. Not sure what has changed in the firmware / envoy back end but I’ve figured out what is going on. My code is quite aggressive: It checks for /production.json every second _but_ has a 1 second no response timeout. This is normally ok but..
Also, every 3 minutes, I check for /inventory.json. It seems that sometimes /inventory.json is taking more than 1 second to respond with data (sometimes takes 600ms, sometimes takes up to 6 seconds) and is “blocking” so other requests get queued / delayed.
My code then, at the next second, requests /production.json but as /inventory.json is still processing – it times out, then requests it again a second later.
It seems that prior requests get queued still even though the TCP connection has been ended [RST, ACK (forcefully ended)] with the timeout. This ends up running away and the envoy appears to have locked up. If you stop _all_ requests and leave it for about 10-20 minutes, it recovers.
You can reproduce this problem by loading up one of the .json files in your browser and constantly spamming the Refresh -> Stop -> Refresh -> Stop button (without waiting for the page to load each time). After about 20 or so goes at that you should see that your final request and ones shortly after take quite a long time to be responded to by the envoy.
Something has changed with the way the envoy handles requests or how quickly it can respond to requests.. compounded by my aggressive code.
I guess I now need to code, my side, failure detection and make my code “back off” so that the envoy can recover or fulfil all pending requests (double the delay until the next request, then double it again if it fails a second, third, fourth (etc.) time).
I hope that the new firmware, with this quirk, does at least fix the total lockups that required a power cycle to recover from :D
No crash or lockup since the firmware update!
Yay. I no longer have to power cycle the envoy weekly.
Thanks for your work on this! I wish that I had an Envoy-S. My two older Envoys don’t have many of the items mentioned. In case you are interested, I created a Python script to archive the event log on the older Envoy version — it may work with your Envoy-S… Short link: https://goo.gl/kWHcBJ
Hi,
I can get a fair bit of data following you instructions including panel production, but what program did you use to make your own system and refresh the data?
Thanks
I’m using asp classic and built in logic to cache the response for X seconds before making another request.
This means I can have multiple displays or systems (automatic space heater that uses up spare energy instead of exporting) polling my ASP page multiple times per second! but my ASP only “bothers” the Envoy-S every 3 seconds and returns that cached result until it expires.
See my previous comments (above) about how the Envoy really doesn’t cope well with lots of requests in quick succession.
ASP Classic was my choice because I knew I could develop something rapidly. Anything else would also work fine, PHP, Perl, Python or even a simple bash script using wget and grep.
Thanks for the quick response. Any chance you would be able to share the script/programming you wrote for your own system which I could then adapt. I’m not great with writing my own. Also I would probably only request every 10 seconds so I don’t upset my envoy.
Thanks again.
Difficult to share without knowing what it is going to be used for. My code spits out javascript which probably won’t be suitable for whatever you want to use it for. And my code is ASP classic which requires an IIS / Windows server.
No worries. I’ll work on it from another angle.
Thanks.
Ken’s python script does not work with the current envoy-s firmware. It gets both an unauthorized error, and it looks like the envoy login doesn’t allow access to the event log, at least at that location, even with the envoy:last6ofsn login.
Thanks for checking that Dave. You can try downloading the event log directly to see if the location is correct — it is at — http://%5Benvoy ip address]//datatab/event_dt.rb
I wish that I had an Envoy-S to test with. Does anyone know if the Envoy-S will work with the older M190, M215 and M250 series inverters?
The fact that it exposes the production information makes it very attractive to me…
Compatible Micro-inverters.. a convenient table is here:
https://enphase.com/en-uk/products-and-services/envoy/family
Looks like the M215 and M250, yes.. M190 isn’t even mentioned.
On the Envoy-S you need to issue the event log request with start= and length= variables for it to return information. For example /datatab/event_dt.rb?start=0&length=153
I’m still stuck on the reverse engineering of the password hashing function :( Shame.
Off-topic: I can find at least 688 Envoys just “open on the internet”.. I do hope they’ve been designed securely! Otherwise another potential IoT disaster.
Yes, the whole IoT thing is a disaster waiting to happen. My two Envoy(s) are behind multiple firewalls.
Since I don’t have the “S” model, I can’t do any testing. My Python script is fully commented so you can see where the request to the event_dt.rb has a whole bunch of parameters being passed. I do understand about the start and length on your “S” version.
I wish you luck with the password hashing. It is a shame they don’t expose the data with a password known to the user.
BTW, to get the parameters I did the
The keys were stripped out: control and shift and q
Guys if I want to pass my user id envoy and password – last six digits of the Envoy-s to read the individual inverter / panel data – does anyone know the format of how I need to pass these parameters? My code has issues passing these credentials.
If I key the local device URL into Chrome 192.168.0.12/api/v1/production/inverters and enter envoy and 012345 when it prompts for user id and password it accepts this and shows all the relevant details:
{
“serialNumber”: “121440006251”,
“lastReportDate”: 1484971459,
“devType”: 1,
“lastReportWatts”: 172,
“maxReportWatts”: 255
},
{
“serialNumber”: “121440009849”,
“lastReportDate”: 1484971423,
“devType”: 1,
“lastReportWatts”: 220,
“maxReportWatts”: 251
},
etc
My html file is executing the following JSON code
$.getJSON(“http://envoy:012345@192.168.0.12/api/v1/production/inverters”, function(data) { …}
BTW – is the code for the screen on the first page shareable? Love to see how youo did what you have!
Many thanks,
Matthew
It looks like getJSON might be jquery? If so it seems that jQuery can’t do Digest authentication and will likely just be sending Basic authentication headers.
http://stackoverflow.com/questions/5288150/is-digest-authentication-possible-with-jquery
This looks like an addon for jquery that might help:
https://www.openhub.net/p/digestj
(I’m no expert in jquery so the above might be wrong or out of date).
Ref – the code from the first screen shot.. it is in ASP Classic which pretty much nobody uses (or should be using) so probably not worth sharing.
http://trilema.com/2017/the-incidental-humiliation-of-obamas-clean-energy-policies-marc-andreessens-internet-of-farts-and-other-such-comedic-gold-brick
This guy hangs out with people who attempt to factor RSA keys. It appears that early Envoys all use the same SSH RSA hostkey from Debian – and it isn’t prime!
Interesting article you link to – doesn’t seem to be the specific one about the keys used but does have some interesting stuff in it.
Seems that they are looking at a tsv file of some energy generated (doesn’t say source or what other info the file contains) but have deduced that there isn’t much energy being produced vs. the energy to manufacture solar generation kit.
A bit amazed by their findings / calculation. The stuff on my roof is certainly more than covering my daily energy usage and I’m exporting more than I use. I’m not sure on the energy cost to produce the equipment in the first place but I can’t believe the guys claim that they are not economical. “To recap : “solar power” costs more in energy alone to merely produce the hardware (we’re not counting the minerals, just the energy to assemble all the stuff together) than it will ever output over its lifetime.”
“For every Watt of energy an installed solar panel ever produces, it will have consumed about three Watts of energy. You are actually saving two Watts by not putting one in. ”
I really don’t believe that but I’ve yet to investigate their calculations in depth.
Yeah if they are correct with their calculations then whoever installed that solar array did a poor job. My array is covering my energy use as well, although admittedly it will still take quite awhile to pay off. I’m using my excess electricity to mine cryptocurrencies which helps.
The more interesting finding to me is the link to the SSH RSA key. I figured out how to extract and convert the key on my Enovy-S and it is not the same as what these guys claim to have found. So it appears this is a problem with old Envoys, not new ones.
Hi jam… you said earlier:
I currently have OpenHAB polling my envoy-s production.json and logging production and consumption data, and also activating heating etc based on available power! :)
Can you expand on this please? Their doco is confusing me — I don’t see anywhere where it can take input data and graph (or do something) with it?
That’s right, you won’t find anything specific in the docs about it. But all the information is actually there – you want to poll your envoy, not use the enphase binding.
So I just use the http binding to poll the envoy-s and parse the json returned. You parse out the solar production and consumption values into an openhab “Number” variable. Then just switch on the rrd4j “persistence” for those Number variables.
Then generate a chart from the rrd4j. Or action what you will based on the current power produced using the rules engine.
http://docs.openhab.org/addons/bindings/http1/readme.html
Ta for the info. Yeah, I don’t trust the Enphase cloud API… I have suspicions that they’re munging data.
I’m putting together some Python, using SQLAlchemy to pull all data from the Envoy, storing it locally… and will probably end up graphing it using plot.ly and Flask as the frontend.
I’m stoked as well, because production/inverters is working for me with creds as suggested above ;) so that’ll be interesting to see which panels do what, at which times of day.
Thanks for all the good work here! I upgraded one of my Envoy(s) to the “s” model. That has allowed me to develop some Python for it. I created a repository for the Inventory script on GitHub today. A post with that link and a screenshot can be seen at: http://www.kenclifton.com/wordpress/2017/06/envoy-s-local-solar-inverter-inventory-and-status/
As well as running on standard Python version 3, it also works in the Pythonista App on iOS.
I have found the following code gets a lot further when I execute $.getJSON(“http://192.168.0.12/api/v1/production/inverters?locale=en&user=envoy&password=012345”, function(data) {
Now that it seems to pass authentication I am encountering network error
SCRIPT7002: XMLHttpRequest: Network Error 0x2efd, Could not complete the operation due to error 00002efd.
I found this on both firefox and edge – on Chrome I recieve
‘Access-Control-Allow-Origin’ header is present on the requested resource. Origin ‘http://127.0.0.1:8000’ is therefore not allowed access. The response had HTTP status code 401.
Access-Control-Allow-Origin is going to scupper you there. The envoy must be replying with that in the HTTP headers which then tells the browser to not allow inclusion of that data into your json / javascript.
The only way around it is to get the envoy not to provide that header.
If I were having this problem I’d build a simple php or aspx script to fetch the /inverters data and then spit it out again without the Access-Control-Allow-Origin header!.. or maybe make a scheduled task or crontab to wget the data into a file and then host that file on your own web server.
Well I made a lot more progress by changing @.getJSON to $.getScript – it invokes the request user name and password and when supplied it does return success – and I can see the data returned to it in debug mode, the only issue I have is I log all the variable to console.log and somehow the data variable itself comes back as undefined. This may be becuase $.getScript is a ayschronous call – and by the time I log the headers, call success etc the data itself hasn’t yet been returned. So it seems a less challenging problem to tackle. In debug mode I can see a Get call is issued – all the headers and Params look correct and the Call stack shows 31 individual inverters data.
My logic is:
https://code.jquery.com/jquery-3.1.1.min.js
table td.box {
width: 250px;
border-left: 1px solid #000000;
border-right: 1px solid #000000;
}
table td.box div {
background-color: #0000FF;
}
Solar Test
$(function() {
var url = “http://192.168.0.12/api/v1/production/inverters”;
$.getScript ( url, function( data, textStatus, jqxhr ) {
console.log( textStatus );
console.log( jqxhr.status );
console.log( data );
console.log(“Load was performed”);
var table = $(“Serial NumberLast Report DateStatusLastMax”).addClass(“table table-striped”);
var totalLastReportWatts = 0;
var totalMaxReportWatts = 0;
$.each( data, function(index, item) {
console.log( index );
totalLastReportWatts += item.lastReportWatts;
totalMaxReportWatts += item.maxReportWatts;
var percent = ((item.lastReportWatts / item.maxReportWatts) * 100);
var row = $(“”).html(“” + item.serialNumber + “” + new Date(item.lastReportDate) + ” ” + item.lastReportWatts + “” + item.maxReportWatts+ “”);
table.append(row);
});
var totalsRow = $(“”).html(“” + totalLastReportWatts + “” + totalMaxReportWatts + “”);
table.append(totalsRow);
$(“#results”).append(table);
});
});
Any idea how to get the data to be actually linked to the data field so I can parse it correctly?
I also tried re-structuring my code to execute the GetScript first – which toggles the Enphase Envoy-s unit to ask for authentication via a pop up window in the browser running the html file. Once these details are passed the call progresses – but with the returned data being classed as undefined and failing the string parsing.
But I then added a second GetJSON call immediately after the GetScript call to see if the Authentication details created for the Get Script call where inherited by the subsequent GetJSON call – unfortunately they are not and a CORS error is recorded, buts its a 401 error – failed authentication.
In firefox Debugger – it appears the GetScript call is a JS call type – which fails with a 401 – but the credentials when entered into the pop up Window that appears then immediately causes the Envoy to return the correct data with a 200 status code.
When the GetJSON call is launched it is a JS XHR request and it doesn’t trigger the credentials logic the GetScript call does, so it doesn’t get the required authentication token / credentials and it simply fails with a 401 status.
* * * * * *
I wonder if there is any way to have the GetJSON call trigger the Envoy to ask / launch the credentials pop up window the GetScript did – so it too could pass authentication?
Hey Jam, if you have got it all working could you please share your code somewhere!
Hey there,
Well, I don’t like reinventing the wheel so as I said I used OpenHAB, and the http binding included.
So in the openhab.cfg I set the http binding timeout to 3000ms:
############################### HTTP Binding ##########################################
#
# timeout in milliseconds for the http requests (optional, defaults to 5000)
http:timeout=3000
Then in the OpenHAB “Items” configuration file I make 3 items, one to hold the entire json from the envoy+s, and then one for production and one for consumption:
String Enphase_JSON “JSON data from Enphase Envoy-S” {http=”<[http://192.168.0.4/production.json:30000:REGEX((.*))]"}
Number Power_Consumption (Solar2)
Number Solar_Production (Solar2)
Then I make a rule in the OpenHAB “rules” config file to parse the JSON when it is updated in the Enphase_JSON String item, writing the results to the Power_Consumption and Solar_Production items:
rule “Init”
when
Item Enphase_JSON changed
then
var String data = Enphase_JSON.state.toString
var Number production_value = transform(“JSONPATH”,”$.production[?(@.type==eim)][0].wNow”, data)
var Number consumption_value = transform(“JSONPATH”,”$.consumption[?(@.type==eim)][0].wNow”, data)
postUpdate(Solar_Production, production_value as String)
postUpdate(Power_Consumption, consumption_value as String)
end
and _that’s it_. You can then turn on persistence for the two Items to store the production and consumption values over time, graph them, write other rules to switch other openHAB items on and off based on the values, like power switches, etc etc etc… It just works.
The downside is you have to learn OpenHAB, which might be worth it anyway IMHO! :)
Thanks Jam – something new to learn! So are you parsing per panel / micro power – and if so did you use OpenHAB – to pass the Authenctication paramaters envoy + last 6 digits of the envoy to in your case 192.168.0.4/api/v1/production/inverters
Many thanks – given me something new to research!
Nah, I didn’t bother with the per panel data. I didn’t need it for my automation needs, although it would be interesting to graph my split array separately (9 panels NE, 9 panels NW). In reality though it actually doesn’t vary that much!!
I’m anyway lucky enough to have access to enlighten manager to scratch the per panel itch, and besides if you’re too OCD, per panel data can drive you nuts. 1W difference here, 2W there.. it doesn’t really matter to be honest.
I did python script which scrapes the envoy.local APi’s and combines from different json responses one json (the data I wanted) I use this to store to my db and show it on my magic mirror. Codes are free to use in case it helps somebody: https://github.com/teemuhirsikangas/magicaespeculo/blob/master/scripts/send_envoy.py
I have an Envoy-S and a couple of Enphase Batteries. I am retrieving the data every 2 seconds to display in an C# program and that works fine. However, the battery data reported by the Envoy-S changes only once every 5 minutes; any suggestions how I can get better LIVE data? Some documentation suggested that the batteries need internet connectivity, but sofar I have not been able to find which (WI-FI) IP address it uses, hoping that I could listen in on the conversation. I am not convinced the battery uses the internet.
Someone informed me that the Envoy-s uses digest authentication – which isn’t avaliable in json directly – but you can code a AJAX call for digest authentication – which succeeds in triggering the same authentication logic that a browser encounters when requesting the inverter data. So I am a big step closer – in debug mode I can see the authentication passed and correct data returned – but the returned data isn’t being linked into the call – so I can’t parse it – not sure where it is going!
The AJAX auth code is simply:
$(‘#importButton’).click( function() {
var commCareFormURL = $(‘#url’).val();
var commCarePassword = $(‘#password’).val();
var commCareUserID = $(‘#user’).val();
$.ajax({
type: “GET”,
xhrFields:
{
withCredentials: true
},
dataType: “jsonp”,
crossDomain: true,
url: commCareFormURL,
async: false,
headers: {
‘Authorization’: “Basic ” + btoa(commCareUserID + “:” + commCarePassword)
},
success: function (jsonData) {
console.log(“AJAX Get Succeeded”);
var table = $(“Serial NumberLast Report DateStatusLastMax”).addClass(“table table-striped”);
var totalLastReportWatts = 0;
var totalMaxReportWatts = 0;
$.each( jsonData, function(index, item) {
console.log( index );
totalLastReportWatts += item.lastReportWatts;
totalMaxReportWatts += item.maxReportWatts;
var percent = ((item.lastReportWatts / item.maxReportWatts) * 100);
var row = $(“”).html(“” + item.serialNumber + “” + new Date(item.lastReportDate) + ” ” + item.lastReportWatts + “” + item.maxReportWatts+ “”);
table.append(row);
});
//
var totalsRow = $(“”).html(“” + totalLastReportWatts + “” + totalMaxReportWatts + “”);
table.append(totalsRow);
$(“#results”).append(table);
console.log(jsonData);
},
error: function (request, textStatus, errorThrown) {
console.log(“AJAX Get failed”);
console.log(request.responseText);
console.log(textStatus);
console.log(errorThrown);
}
});
});
Trouble is although the authentication is launched – neither the success: nor the fail path seems to be triggered!
hi clever people!
i’m not a coder, so have no idea the feasibility of what i’m about to ask.
not sure if you’ve seen it, but Mozilla are heading down the path of creating a standards-based approach to iot, or “web-of-things” as they’re calling it. the just released the first version of an open extensible gateway – https://iot.mozilla.org/
I’ve been having a bit of a play, and was thinking it would be nice to include info from my enphase gateway there as a “thing”
As the webthings API is published (https://iot.mozilla.org/wot/) looks very much like it’s based on JSON, as is the “production.json” file on the enphase gateway, how hard would it be to massage the data from enphase into an “adapter” for the mozilla gateway?
Again, i have no idea the complexity of what i’m asking to apologies if it’s a crazy question!
cheers
dan
Many thanks: /production.json is the business!
Rgds
Damon
…and here’s some of the output, graphed!
http://www.earth.org.uk/electricity-storage-whole-household-2018.html#Data
Thanks again!
Damon
Pingback: What is inside the Enphase Envoy-S (teardown) | thecomputerperson
Hi Damon and everyone else.
You might find this page of interest.. I managed to dismantle and photograph the insides of the Envoy-S Metered:
\o/
Mmmm, electronic-y guts with magic smoke still in, mmmm!
Rgds
Damon
BTW, I just read a comment elsewhere that Enphase originally intended to bring out a battery version with 2 inverters so twice the charge/discharge power. Presumably at the cost of some battery life, but that could be worthwhile, especially given a mix of types in the system to balance the wear.
Do you know any more about that?
Rgds
Damon
Not a clue. I’ve not researched battery storage and I’ve only ever seen an Enphase Micro PV system twinned with a Tesla battery system.
Damon – that is correct – but it never launched. The great news from their investors tech conference in the IQ8 a 440W inverter will be in both their batteries Encharge and roof inverters. Better still the inverter is grid agnostic (islands from a fault gird – google Enphase Ensemble) and the battery is now a 3.3 kWh modular unit with 3 x IQ8 inverters giving a usable 1.3 kW sustained output. The unit will also be sold as a 3 x 3 = 10 kWh unit with a 3.8 kW sustained output. Launch is in H1 2019 – but I read they are targeting Africa and India first. – Matthew
Ah – tried it on my mobile – works fine thank you!
Bingo.. the app does need to be run on a mobile (or possibly “Bluestacks” android emulator for windows) as it contains the Enphase ARM based file that won’t run on Windows / “x86” computers.
I am back on my quest to try an get an Excel sheet to poll the Enphase Envoy-s on my network using a web IP access call to the old (classic=1) Production page. So I am punching out to http://192.168.0.12/home?classic=1 – but even in my browser trying to select the Production page invokes a user id / password challenge that I can’t get past – even with the installer id.
When querying the local envoy – using the classic=1 flag to get the old interface on your web browser – does anyone know the password to access the Production page?
Just use the json? http://192.168.0.12/production.json ?
And what installer password are you trying to use? On newer Envoy-s’s it isn’t likely to be the last digits of the serial number. It is a unique hashed value that you would need to generate using the app on this page:
Trouble is – Excel hates trying to ingest json and it used to work beautifully from Excel when it pointed you to an already coded html file. That is the so annoying factor. The page is already there but some moron password protected i and refuses to give out the password!
Interestingly today – Enphase Australia tried to recreate this behaviour on their test rig and it didn’t ask for authentication. I asked and two people on the Australian Enphase User group quickly tested it and confirmed the issue occurs for them. Do folk in America and overseas see the same behaviour I wonder? Wonder if this is how the anomaly slipped through – maybe it doesn’t occur in Enphases test rigs?
Works instantly for me in Excel.. https://imgur.com/a/9hNOMkQ
Wow that is impressive! I only have Excel2013 – which acts very differently when you try and get production.json – asks if you want to open the file – then says unsupported file type and dies. Yours I see opens Power Query – that video was great. I may see about getting an upgrade to office 365 (which we use at work). If it can import .json and format it easily I could side step a lot of problems. Is the query tool you used in that video native to Excel office 365 – on another add in one must buy? Many thanks!
Native. Office 2016 would also do (if you would rather buy than rent / subscribe).
So I just downloaded Power Query and used it in Excel 2013 – and it worked very similar to your demo. Many thanks mate! Two questions for you: Can you run timed queries – i.e. say every five minute run this query using PowerQuery or do you sue Data -> Connections and set refresh to say 5 minutes? Secondly can you use this to get api/v1/production/inverters data – when I try to pass user id envoy and password last six digits of my envoy – it fails – I presume it is doing simple authentication and the envoy-s may wish digest authentication? Best regards
Looks like it is in the connection properties.
http://www.excel2013.info/power-query/automatic-update/
I’m no expert in excel :) Just googled for that one.
So this is starting to give me in Excel what I – the only data morsel I wish to read if possible is total daily grid consumption.
I can’t seem to find where or if the local Envoy-s hold this? So far now I can easily read in 1 minute intervals – current production, consumption, net power (import/export), Total watts produced today, Daily total Consumption (Grid import + Solar self consumption).
I really want in real time grid daily consumption. So I either need this filed or need to know Total Solar Produced daily- less total export daily = Total daily self consumed to figure out total daily grid import, or read directly total daily grid consumed. Anyone know if this is provided anywhere? Thanks
So Enphase told me to access the original view classic=1 of the new envoy-s the credentials are enphase and a password that looks like a Windows install key. Trying to find out if that is a generic key I can share with all or only the key to my device. Works well – also works for the api/v1/production/inverters.
Anyone know how in Excel 2013 with Power Query I execute a Web call and tell it the web access setting to use – I can’t figure out how to bring up the page that asks what time of web call and what the user id and password is!
I still haven’t found out two things.
The first – how or if the envoy-s stores net export or net grid import – there is a data element for it – but at least on my machine it is always zerol! Using production.json noting watt hours in net consumption is always zero:
type “eim”
activeCount 1
measurementType “net-consumption”
readingTime 1537632361
wNow 785.029
whLifetime 16673757.994
varhLeadLifetime 17297867.514
varhLagLifetime 182755.155
vahLifetime 40903738.501
rmsCurrent 5.712
rmsVoltage 243.594
reactPwr -1066.64
apprntPwr 1391.662
pwrFactor 0.57
whToday 0
whLastSevenDays 0
vahToday 0
varhLeadToday 0
varhLagToday 0
The second is to determine if it is possible for PowerQuery to pass authorisation credentials to the production/inverters URL. I have tried specifying an advanced call to the URL and adding header Authorisation with data “Basic ZW5waGFzZTpmZTlmMzQzNzU0YmQxZjRkYTU4NGQzNmMwNmI5YTU4Mg==” with is the base64 coded “enphase;password” that works for me from enphase. But the call says anonymous has to be supported.
So I tried a new approach with a lot more more success. I created a Execl file with two worksheets. In the first worksheet I wrote a normal Data -> web query that polls the website, successfully passes the credentials – grabs the inverter JSON results and simply writes it to a local file called inverters.json. Then in another worksheet write a power builder query to read this inverters.json file and extract the inverter data – scheduled to run every minute. This worked well!
The only challenge I have now is storing the first procedure call and setting its properties to run every minute. Excel doesn’t seem to want to store this call if Excel isn’t trying to do something with the results. Any suggestion guys?
@Matthew: thanks for the heads-up re IQ8 / Ensemble!
Rgds
Damon
I’ve installed an Envoy-S and a 3 phase system and was interested in high resolution logging (ie: every few seconds) of meter data – particularly AC voltage and current.
Unfortunately the JSON file at http://envoy/production.json just shows the sum total of all 3 phases as the “rmsVoltage” (so, in my case, about 750 V). Same deal with current. Not useful at all.
However, using the installer toolkit app – or the password protected web pages on the Envoy (discussed in “Reverse Engineering the Enphase Installer Toolkit”), it is possible to see all these values, updated every second.
So I did some packet sniffing and found another JSON file:
– http://envoy/stream/meter
Containing:
data:
{"production":
{"ph-a":
{"p":-.0,
"q":129.13,
"s":148.351,
"v":251.904,
"i":0.589,
"pf":0.0,
"f":50.06},
This is just for the production meter and the first phase (“ph-a”). It continues on for each phase and for each meter.
Unlike http://envoy/production.json, this requires installer authentication. But since it is a continuous data stream, I guess it’s probably a lot less taxing on the little Envoy than a new HTTP request every second.
More importantly, it has the info I wanted. Plus reactive power and frequency.
I can access it from the Linux command line with: “curl –anyauth –user installer:******** http://envoy/stream/meter“.
Password is the installer password unique to the envoy, thanks to your awesome app at https://www.dropbox.com/s/csmm9izuu5nyllj/AndroidXam.AndroidXam.signed.apk?dl=0.
Note that the first curly bracket is missing from the “data” key, so you need to add that bracket and the quotation marks before the JSON can be parsed.
Hope this helps someone.
Hi Stuart,
Fantastic find with the hxxp://x.x.x.x/stream/meter URL! I will add that to the page. I don’t have 3 phase so didn’t even consider that it might be a problem for some people.
Using the wealth of information here I’ve put together a small python script that scrapes data from my envoy and makes it available to prometheus (a time-series database) and grafana (a service for plotting data).
Everything runs in docker and it’s pretty trivial to set up once you have calculated your installer password.
Everything can be found here: https://github.com/petercable/solar-observatory
Hi Stuart, totally out of my depth so far. 2 days into this.
I have my installer password, and it works for http://envoy/installer/setup/home#overview etc, but if I try http://envoy/stream/meter it comes up with http://envoy/error and page not found.
There’s some basic information displayed on the page (serial number etc) and a simple menu. Anything “advanced” I click on asks for authorisation, but username installer and password don’t work. http://envoy/home?locale=en&classic=1 shows if I hit the home, but neither username installer or envoy and the respective passwords work to go furher
If I change to classic=0 (or remove &classic altogether, I get in further.
I have:-
Software Version D4.5.79 (08d137)
Software Build Date 29 Mar, 2017 1:38 AM
3 Phase system, 6 x batteries, sadly my solar was installed before and I’ve got a single Fronius inverter.
This is how to access the inverters page in Python 3. Replace 123456 with the last six digits of the serial nr.
from requests.auth import HTTPDigestAuth
url = “http://envoy:123456@192.168.178.13/api/v1/production/inverters”
response = requests.get(url, auth=HTTPDigestAuth(‘envoy’, ‘123456’))
print (response.json())
Thanks for the password generator , do you have any program for data logging / charting directly from the envoy-s available that could be used on a Linux distro ?
https://github.com/petercable/solar-observatory Thanks pete
Thank you to everyone that has commented and the excellent work done by the author. Shame we have to go to this length of reverse engineering / scraping when Enphase are collecting “our” data.
I’m using the API call “http://envoy/api/v1/production/inverters” to interogate the individual power production of the inverters. I’ve noticed they only update every 15 minutes, is this what others are observing?
Thanks
Paul
Thanks all! – very useful I now have my system now scraping the /stream/meter data.
I have 3 phase power, so it’s great to see that each phase is doing. I’m hoping this may allow me to identify particular loads, i.e. the dishwasher, dryer, pool pump etc…
It took me a while to be able to read the /stream/meter data, I ended up authenticating to /installer/setup/home first. Then calling /stream/meter (using the same HTTP session).
Regrading the JSON data format this is consistent with a stream of Server Sent Events (SSE). I used the following python library to deal with this:
https://pypi.org/project/sseclient-py/
So no modifying of the payload required.
@Paul – yes my inverter data only updates around every 15 minutes also.
@Hamish – how did you maintain session information between calls? Did you use curl from the CLI or Python? The curl example given further up by @Stuart (“curl –anyauth –user installer:******** http://envoy/stream/meter“) doesn’t seem to work for me, I just get a null response
I have found http://envoy/api/v1/production/inverters can be set to refresh every 5 min ( toolkit needed network-settings standard mode ) and http://envoy/stream/meter is live my gripe is I had no way to remove dead inverters from showing in the output , so I had to alter scraper to disregard dead inverters https://github.com/MasterCATZ/solar-observatory the other is using wifi connection it lags the envoy and network pretty badly
@Cats thanks for the pointer with regard to changing the 15 min interval to 5 min. I had to set mine to Low Bandwidth and then back to Standard for it to start sending every five minutes rather than 15 minutes. Still having no joy with streaming. Is it possible to observe the stream via curl or web browser?
I would like to add my thanks to the writer of this site. I used the information you gathered to write my own tool for displaying data. https://github.com/dlmcpaul/EnphaseCollector
It can output to an influxdb or pvoutput web site for easy visualisation or just use the internal web page to display current overview.
re: whToday being zero in “net-consumption” within production.json. if you’re after a “today” net consumption calculation for home automation, make a reading at midnight against whLifetime and store the value. read it again when needed and subtract the current whLifetime against the midnight whLifetime read.
Could anybody help me writing a python script that can trigger a command when produced power is > 5000w?
I have both my heater and poolpump connected to a raspberry allready and would like to activate then when more then 5000w is generated
Email: geennut@hotmail.com
Quite a different yet related query – any information about the commands Enlighten must be issuing to the Envoy-S? For instance, when I change in Enlighten when I want to charge my batteries from the grid? Maybe my monitor program could then tell the Envoy-S to start charging whenever it is worthwhile. Enlighten only allows for charging between midnight and 6 am, where maybe I want to start charging at 15:00, and perhaps the direct approach will allow that. My power provider charges me spot prices plus a fixed component varying between peak and off-peak times, and my monitor program could decide when it is time to start charging, depending on time of day, current charge in the battery, and current power price. The spot price component sometimes drops down to almost zero – a great time to charge, but has also been very high, even (exception) at $10/kWh – better not charge then!
Thank you so much, it’s perfect for http://local_ip_through_vpn/production.json and the six CT information !
Does anyone know where the firmware update URLs are ? I haven’t been able to gather that information yet , nor have I seen it mentioned ..
Hey @Paul – yes I used python to manage the session
Here’s a code snippet (untested):
import requests
import logging
import sseclient
from requests.auth import HTTPDigestAuth
logger = logging.getLogger(‘pythonStore’)
logger.info(‘Logging in…’)
session = requests.session()
url = ENVOY_URL+’/installer/setup/home’
resp = session.get(url, auth=HTTPDigestAuth(ENVOY_INSTALLER_USER, ENVOY_INSTALLER_PASS))
if resp.status_code != 200:
logger.error(‘Login failed. Error code [%i]’ % resp.status_code)
logger.error(resp.text)
else:
url = ENVOY_URL+’/stream/meter’
logger.info(‘Getting data from [%s]…’ % url)
resp = session.get(url, auth=HTTPDigestAuth(ENVOY_INSTALLER_USER, ENVOY_INSTALLER_PASS), stream=True, timeout=(10, 300))
if resp.status_code == 200:
sclient = sseclient.SSEClient(resp)
logger.info(‘Start processing meter data…’)
for event in sclient.events():
write_influxdb_meter_data(event.data)
logger.info(‘End processing meter data…’)
else:
logger.error(‘Failed to process meter data. Error code [%i]’ % resp.status_code)
logger.error(resp.text)
anyone else getting blocked from accessing http://envoy.fritz.box/api/v1/production/inverters 30 minutes after logging into the system via Toolbox ?
First, thank you (and everyone else here) for sharing all of this info, very helpful to me with an Envoy IQ.
Does anyone know what’s in the Envoy’s database and how to access it? I see the reports of the database space utilization. But what’s in there? I can only guess that it is some metric storage.
After about a month and a half of operation, my Envoy IQ’s db is only 1/2 full.
The event log seems to max out at 500 entries, which is only about 1-2 days worth with 26 panels/microinverters.
Last question, are there any other sites where people are discussing the Envoy and local access?
My Envoy-IQ status page says that the metering isn’t setup. Despite that, I still seem to be getting production data. My envoy couldn’t be located near the panels, so I’m not set up to monitor consumption. I need to find some other solution to handle that which will work with 2 x 200 amp panels.
So if the metering isn’t setup, what’s producing the production data that is being reported?
If the metering was setup correct, what else would I be seeing?
Anyone try to configure metering on your own?
If I try the “Wizard” via the Envoy’s built in web app after logging in as installer, it seems to hang and never complete.
Thanks for any insights
Thanks to the great info provided here I was able to put together a Python based realtime(ish) monitoring dashboard for my PV system. Everything works great, however there is one missing bit of information that I haven’t been able to find.
I need to know how much has been imported from the grid (not ‘consumption’) at any given time of the day (or even total grid imports per day). MyEnlighten seems to know this figure because it reports it in the consumption part of the UI: Approximately xxxx kWh imported from grid.
Does anyone know how this info can be obtained? I’d like to keep track of daily electricity costs/credits. Thanks.
A further complication revolving around knowing the actual grid import figure, is knowing how much energy was actually exported (for FIT calc).
MyEnlighten reports Net_Energy_Exported as a simple calculation: Energy_Produced minus Energy_Consumed. But this isn’t right because when the sun goes down, consumption keeps increasing, while Energy_Exported keeps decreasing!
The total amount of energy exported to the grid for the day should not vary after the sun sets (just like total production does not vary).
This is all tied to the same issue of conflating solar self-consumption with actual consumption through grid imports, into a single ‘consumption’ value.
The only way I can think of getting accurate grid import/export data is by monitoring power generation/consumption over a 24 hr period in 1 second intervals (so that I can work out energy values).
I’m starting to do this now, but I’m not sure how accurate the energy values are as the Envoy hub does not respond to get requests with military regularity. So there will likely be some drift over 24 hrs…
The other problem is that sometimes Envoy stops responding to requests for minutes or not at all and I have to restart my app, but then the running totals are lost (as are gaps in time). Argh!
The envoy is not powerfull enough to respond to requests of that frequency. I find 1m intervals to be accurate enough and have not had any issues with the envoy stopping.
I’ve been able to poll at 1sec intervals for well over 12hrs at a time without issue. But eventually, it does fail with this message:
“A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond”
I have to wonder that if it works fine for such long periods of time (at high frequency), when it fails that it might perhaps have something to do with Enphase querying the server remotely or trawling through the internal DB, or something, which makes it unresponsive to my GET requests…or the Envoy doing some housekeeping itself.
The trouble with reducing the polling rate to 1min is that there is a reduction in accuracy when tracking total energy for grid-in and grid-out purposes.
I’ve found that the failed requests still seem to get processed by the envoy so the more it’s failing and the more you try the longer it I’ll fail for.
With my one I time out after about 10 seconds and then detect that and “back off” for 60 or 120 seconds to give the envoy time to recover and then resume normal 1 second polling again.
>the more it’s failing and the more you try the longer it I’ll fail for.
Yes, that’s what I’ve found. Once it fails, it rarely comes back (sometimes it does), until I stop the app and then restart after a short while.
I’ve tried a few different things, which seem to work (so far). I scrape for ‘production.json’ and ‘production/inverters’ – the former at 1sec intervals, the latter at 20sec (as it doesn’t get updated as often).
Now I’ve switched to 3sec & (still) 20sec intervals, but I’ve also added different user-agents in the header requests. I also back off for 20sec on failure.
So far, this has worked well and I can live with 3s vs 1s polling intervals. Also, I’m just using a ‘dumb’ delay, but might try using the backoff_factor construction.
Doc, I’ve figured how to calculate Import From Grid for about a couple days now, I have it working in Home Assistant scripting language and it’s within 1kw from my Utility provider’s smartmeter readings, after many trials and errors.
you only need to poll http://envoy.local/production.json
it is basically the following rules
if consumption[1][“wNow”]>0
consumption[1][“wNow”] – ( production[0][“wNow”] – production[1][“wNow”] )
if (consumption[1][“wNow”] <=0 then the formula becomes
0 – ( production[0]["wNow"] – production[1]["wNow"] )
if the result of the above formula's is <0
then the realtime Import Power is 0
I take the result of the above convert to WH. Since I sample envoy about every 2 minutes, I multiply each calculation by about (120/3600)
then I take cumulative sum throughout the whole day. When I compare this summation result its slightly larger than myenlighten.com but slightly less than 1kw from the utility website smartmeter readings smartmetertexas.com
I do have a few more cleanup rules, I don't know 100% if they are making any difference.
I do set the readings (wNow) to zero if Powerfactor=0 ,
for production readings I also set (wNow) to zero if powerfactor<=0 or wNow<0
I have 2 additional rules I have to check if they are needed.
I'm still working on the Export rules, just have a few more days to try a few things but I almost have that one figured out as well, for export i'm about 1kw higher than the utility providers smartmeter readings.
My problem now is trying to figure out how to store End of day results from Home Assistant, into a new table or a new History Chart .
>Doc, I’ve figured how to calculate Import From Grid…
>you only need to poll http://envoy.local/production.json
Hi Pierre, yes, that’s where I get my data from too. I poll the inverters only to get individual panel production values which I display in a bar graph.
For the energy calcs, I’m doing something similar to you, but using production[1][‘wNow’], not production[0][‘wNow’]. I don’t really understand the relevance of production[0][‘wNow’] – it has a different timestamp compared with all the other timestamps, so it just seems to be an earlier version of production[1][‘wNow’]…
My calculation is as follows:
# Get elapsed time from last read
timestamp = time.time()
tdelta = timestamp – tsPrevTime
tsPrevTime = timestamp
if consumption[1][‘wNow’] >= 0 :
# Importing from grid
wGridIn = consumption[1][‘wNow’]
whGridIn += (wGridIn * tdelta / 3600)
else:
# Exporting to grid
wGridOut = -consumption[1][‘wNow’]
whGridOut += (wGridOut * tdelta / 3600)
This works pretty well.
> …then I take cumulative sum throughout the whole day.
> When I compare this summation result its slightly larger
> than myenlighten.com but slightly less than 1kw from the
> utility website smartmeter readings
I suspect the reason why you’re off is due to the long-ish sampling rate of 2 mins – a lot of fluctuations can occur over 2 mins which you won’t catch.
Also, don’t assume that because you poll every 2 mins, that you will get a result every 2 min from Envoy. I poll every 3 sec but keep a time delta between polls to use in the energy calculation (it fluctuates a fair bit).
What I have now works okay, but it does have some drawbacks. I have to start the app before sunrise because then I know that the whToday consumption data is definitely from grid usage. Then as the app runs throughout the day, it keeps accumulating the energy import/export as shown above.
The biggest problem is that after many hours of running, the app sometimes locks up and I have to restart. But restarting after sunrise means that I can’t determine real grid in/out figures. I have to figure out why it locks up (it doesn’t appear to be the GET requests…maybe an issue with Python – dunno yet).
Hi Doc,
thanks for the info, I’ll have to try to convert my readings to energy first before using them in the formula. Maybe that’s the source of my issue.
here’s what I know of the differences between production[0][‘wNow’], and production[1][‘wNow’].
when I read production[0][‘wNow’] convert to energy and perform cumulative sum, It gives me the Total Energy my panels produces, yesterdays cumulative sum value matches exactly myenlightens.com Total production to the 2 decimal location they display. Unfortunately envoy updates this data only every 15 minutes. production[1][wNow] is more realtime,
but converting to Energy and performing cumulative sum does not add up to Total production, and its also much less than Exported Energy to grid. Cumulative sum of p1.wNow yesterday was about 24.65KW same as p1.whToday
but my export to Grid was 47.65KW and my total production was 67.76KW
(export to grid is almost 67.76 – 24.65= 43.11 just 4kw discrepancy.
It looks like the solar panels only consider exporting to grid when Total Energy production is above production[1][“wNow”] threshold, provided this power difference can be used to compensate all of your power consumption, and still have enough power left to export to the grid. if we reduced all our consumption to 0 export energy is roughly
production[0][“wNow”] – production[1][“wNow”] , but to get exact Export Energy you need the negative values from consumption[1][“wNow”] as well to add to the exported energy.
the difference of the 2 production values are roughly best case Exportable Energy, don’t know why or where does the production[1][“wNow”] energy goes to. could it be Waste energy to convert DC to AC? don’t know why the negative values in consumption[1][wNow]
are needed to get exact Exported Energy.
my export to Grid formula is below, except in Export to grid calculation
you need to include the negative values in consumption[1][“wNow”] .
I’ll have to try converting the readings to energy 1st then using
the formula below, see if calculation is better.
( production[0][“wNow”] – production[1][“wNow”] ) – consumption[1][“wNow”]
I tried to increase the frequency of measurements today, it caused envoy wifi to be lost for about 15 minutes, homeassistant then was throwing errors, luckily it can survive envoy blackouts or occasional envoy latency in responses. I may have to rewrite this in Java to control envoy request more carefully.
could your app unresponsiveness be related to envoy becoming unresponsive after too many repeated request? In java Depending on how nested packages you’re using are, when an exception occurs in java and many repeatedly it could consume memory faster than Garbage collection can reclaim the memory, then java will crash with out of heap memory. Not too familiar with python though.
Have you tried my java application?
https://github.com/dlmcpaul/EnphaseCollector
Hi David,
pretty nice, like seeing code project from work. I have to brush up on my annotations/lombok/spring/lambda I don’t write as much java, I’m more of a database guy that writes java on the side. I got far enough to get it running in eclipse. I added a few more gauges for exportToGrid and ImportFromGrid that my utility provider seems to be tracking separately. Still trying to figure out how to add trend charts for exportToGrid and importFromGrid.
I did notice the production values are from production[1].wNow instead of production[0].wNow, makes me wonder if envoy for different configuration is giving total production only on production[0].wNow for some people like me, and production[1].wNow for others.
my Envoy up until August 7 was actually mis-configured it was reporting I was using 2x from grid, Solar installer sent someone and found it was set wrong, they changed it.
I’m curious if you go the following URL below if your meter modes are Single or Three phase?
This url http://envoy/installer/setup/home#meters/detail/production says my production meter mode is Single-Phase(L-L), same with the consumption meter mode.
I also have to change the Dockerfile slightly to get openjdk for arm32, I’ll be running the docker in a raspberry pi.
Thanks for sharing the code with all of us.
Unfortunately I do not have access to the installer pages.
I can get the meter information from http://envoy/ivp/meters and that gives me this json
{
“eid”: 704643328,
“state”: “enabled”,
“measurementType”: “production”,
“phaseMode”: “three”,
“phaseCount”: 1,
“meteringStatus”: “normal”,
“statusFlags”: []
},
{
“eid”: 704643584,
“state”: “enabled”,
“measurementType”: “net-consumption”,
“phaseMode”: “three”,
“phaseCount”: 1,
“meteringStatus”: “normal”,
“statusFlags”: []
}
suggesting I am three phase with one phase configured? Which sounds wrong to me as I don’t think my house is using three phase.
I don’t so much read from production[0] or production[1]. I look for the “eim” type with a measuremenType of “production” and for consumption I look for a measuremenType of “total-consumption”
production[0] is generally a simple structure marked as “inverters” and it’s wNow value seems ot be a rounded figure.
But different versions of enphase affect the returned json being scanned.
My code handles D4.2.27 & D4.5.79
What version do you have?
I would like to collect responses from other versions as well to try and learn more. It could be three phase needs to be handled differently.
>my export to Grid was 47.65KW… and my total production was 67.76KW
>(export to grid is almost 67.76 – 24.65= 43.11 just 4kw discrepancy.
That’s almost a ten percent difference, which isn’t insignificant…
I was able to get my polling working for well over 24hrs now. I think my issue was to do with threading, as I had two threads polling for different data at differently scheduled intervals. I’ve simplified it down to a single thread and it’s been working reliably.
So, I was able to compare my calculated grid values against my smartmeter and MyEnlighten. Unfortunately, my meter only displays energy in/out to one decimal place, but even so, I was able to determine that my calcs were accurate to within 0.4%, which I think is pretty good.
>could your app unresponsiveness be related to envoy becoming
>unresponsive after too many repeated request?
I think I’ve overcome that problem, as mentioned above, by rethinking my use of threads. The current setup works without issue with 3sec polling of ‘production.json’ and 20sec polling of ‘production/inverters’ (during daylight hours only). So polling frequency wasn’t really the problem as such.
Hi David,
my software information using the url http://envoy.local/info.xml in the device section it shows it is R4.10.35
the meters url you gave it shows I have a very different configuration than yours, split mode with 2 phases for production/consumption and in the consumption section mine is total-consumption yours is net-consumption.
http://envoy/ivp/meters
[
{
“eid”: 704643328,
“state”: “enabled”,
“measurementType”: “production”,
“phaseMode”: “split”,
“phaseCount”: 2,
“meteringStatus”: “normal”,
“statusFlags”: []
},
{
“eid”: 704643584,
“state”: “enabled”,
“measurementType”: “total-consumption”,
“phaseMode”: “split”,
“phaseCount”: 2,
“meteringStatus”: “normal”,
“statusFlags”: []
}
]
when my panels are at full power like below The production value under type “eim” by itself is useless. the java code method
Optional inv = system.getProduction().getInverter();
in place of
Optional eim = system.getProduction().getProductionEim();
does resolve the issue for getting total production. no issue with consumption section of the java code with my envoy software version, i think consumption[0].wNow under mine and your envoy software are reporting similar data, probably in mine consumption[1].wNow will be different. my consumption[1].wNow becomes negative when I’m exporting to the grid, when my panels are not producing any energy consumption[1].wNow is identical to consumption[0].wNow, when my panels are generating some energy but not enough to export my consumption[1].wNow is a smaller positive value.
“production”: [
{
“type”: “inverters”,
“activeCount”: 41,
“readingTime”: 1570378963,
“wNow”: 9405,
“whLifetime”: 22689789
},
{
“type”: “eim”,
“activeCount”: 1,
“measurementType”: “production”,
“readingTime”: 1570378983,
“wNow”: 3545.69,
“whLifetime”: 13784731.903,
“varhLeadLifetime”: 134.344,
“varhLagLifetime”: 5598841.857,
“vahLifetime”: 17567742.118,
“rmsCurrent”: 28.839,
“rmsVoltage”: 249.05,
“reactPwr”: 334.577,
“apprntPwr”: 3591.212,
“pwrFactor”: 0.99,
“whToday”: 7474.903,
“whLastSevenDays”: 165138.903,
“vahToday”: 10206.118,
“varhLeadToday”: 0.344,
“varhLagToday”: 3535.857
}
],
You can also use: http://envoy.local/ivp/meters/readings to get the individual phases.
Interesting, I might switch to the Inverter wNow values then. On my system they track the same except the inverter value was just a little higher and looked to be rounded.
Would you be able to send me some outputs like the ones listed here https://github.com/dlmcpaul/EnphaseCollector/tree/master/src/test/resources/stubs/D4.5.79
Hi David,
just a caution on production[0][“wNow”] of type “inverters” on my envoy setup I have no choice but to use it to get total production that matches myentune.com. In the morning after panels start to generate power and start ramping up, production[0] type “inverter” is only updated every 15 minutes, this includes readingTime, wNow, whLifetime. the wNow value also has no decimal values. The value I get is very close to the sum of all the inverters lastReportWatts from the url below, (lastReportWatts also has no decimal values), the sum of WH calculated values I get is identical to myentune.com.
http://192.168.1.22/api/v1/production/inverters.
Even this URLS http://envoy.local/ivp/meters/readings readings for production is not my total production.
Some of you maybe luckier to get production[1] type “eim” and get total production with greater precision than me, but I can’t be sure of that. Only way to be certain is cumulative sum of production in WH and compare against myentune.com or your provider website if they give you access.
I ran curl and generated .json files for the url you have in your .json files. I’ve uploaded my files to dropbox here’s the link.
https://www.dropbox.com/sh/bhovqkz7i92o4d5/AAAJCIZmsbMcRUWIFSdTrHS-a?dl=0
I’m still not sure if I have Envoy-S or IQ-Envoy, i managed to get the installer password following reverse-engineering-the-enphase-installer-toolkit, but near my unit there is a sticker it says IQ-Envoy. I’ve sent an email to the installer to find out which one they installed.
info.xml does give a part number 800-00555-r03, I googled it, no hit which Envoy it is.
If you want to access data rapidly, consider using http://%s/stream/meter ? This doesn’t tax the envoy.
See Stuart Graves post November 27, 2018 at 21:04 for more details of the stream.
The code below is loosely based on one of the examples that followed Stuarts post, and sums kW for the first 2 phases (ph-a + ph-b) from an envoy-s.
You can use the Python code below in IDLE to stream live data to the console window.
Production data appear to be “live” and change between each value streamed (about every second).
I am using this for demand management.
import os
import time
import json
import ijson
import requests
import threading
from requests.auth import HTTPDigestAuth
#envoy-s host IP
host = ‘#######’
#envoy
password = ‘#######’
user = ‘installer’
auth = HTTPDigestAuth(user, password)
marker = b’data: ‘
def scrape_stream():
while True:
try:
url = ‘http://%s/stream/meter’ % host
stream = requests.get(url, auth=auth, stream=True, timeout=5)
for line in stream.iter_lines():
if line.startswith(marker):
data = json.loads(line.replace(marker, b”))
prod=data[‘production’][‘ph-a’][‘p’]+data[‘production’][‘ph-b’][‘p’]
cons=data[‘total-consumption’][‘ph-a’][‘p’]+data[‘total-consumption’][‘ph-b’][‘p’]
net=data[‘net-consumption’][‘ph-a’][‘p’]+data[‘net-consumption’][‘ph-b’][‘p’]
print(f”{data[‘total-consumption’][‘ph-a’][‘v’] :5.0f}v {data[‘total-consumption’][‘ph-b’][‘v’] :5.0f}v {data[‘total-consumption’][‘ph-a’][‘i’] :5.1f}a {data[‘total-consumption’][‘ph-b’][‘i’] :5.1f}a POWER prod: {prod:6.0f}W cons: {cons:6.0f}W net: {net:6.0f}”)
except requests.exceptions.RequestException as e:
print(‘Exception fetching stream data: %s’ % e)
def main():
stream_thread = threading.Thread(target=scrape_stream)
stream_thread.setDaemon(True)
stream_thread.start()
if __name__ == ‘__main__’:
main()
with my envoy setup the data from http://%s/stream/meter does not give me not total production
here’s production.json production[0].wNow=4161 W
production[1].wNow=1070 W
data from http://%s/stream/meter within 5 seconds of production.json readings
is closer to production[1].wNow
from ph-a ph-b of p (true power) 547.661+546.063=1093.724 W
–production.json–
“production”: [
{
“type”: “inverters”,
“activeCount”: 41,
“readingTime”: 1570641497,
“wNow”: 4161,
“whLifetime”: 22894013
},
{
“type”: “eim”,
“activeCount”: 1,
“measurementType”: “production”,
“readingTime”: 1570641915,
“wNow”: 1070.227,
“whLifetime”: 13859282.806,
“varhLeadLifetime”: 134.344,
“varhLagLifetime”: 5621436.782,
“vahLifetime”: 17657166.059,
“rmsCurrent”: 9.226,
“rmsVoltage”: 246.715,
“reactPwr”: 310.49,
“apprntPwr”: 1136.716,
“pwrFactor”: 0.94,
“whToday”: 4464.806,
“whLastSevenDays”: 170624.806,
“vahToday”: 7532.059,
“varhLeadToday”: 0.344,
“varhLagToday”: 3826.782
}
],
— /stream/meter —
data: {“production”:{“ph-a”:{“p”:547.661,”q”:155.899,”s”:580.791,”v”:123.724,”i”:4.703,”pf”:0.94,”f”:60.0},”ph-b”:{“p”:546.063,”q”:154.276,”s”:579.582,”v”:122.936,”i”:4.722,”pf”:0.94,”f”:60.0},”ph-c”:{“p”:0.0,”q”:0.0,”s”:0.0,”v”:0.0,”i”:0.0,”pf”:0.0,”f”:0.0}},”net-consumption”:{“ph-a”:{“p”:-355.692,”q”:24.248,”s”:261.905,”v”:123.769,”i”:2.116,”pf”:-1.0,”f”:60.0},”ph-b”:{“p”:36.513,”q”:31.768,”s”:87.554,”v”:122.965,”i”:0.712,”pf”:0.42,”f”:60.0},”ph-c”:{“p”:0.0,”q”:0.0,”s”:0.0,”v”:0.0,”i”:0.0,”pf”:0.0,”f”:0.0}},”total-consumption”:{“ph-a”:{“p”:191.969,”q”:-131.65,”s”:320.666,”v”:123.814,”i”:2.587,”pf”:0.59,”f”:60.0},”ph-b”:{“p”:582.576,”q”:-122.507,”s”:668.322,”v”:122.994,”i”:5.434,”pf”:0.86,”f”:60.0},”ph-c”:{“p”:0.0,”q”:0.0,”s”:0.0,”v”:0.0,”i”:0.0,”pf”:0.0,”f”:0.0}}}
With the data that’s available that’s now available via the envoy – is there any reason to pay for Enlighten Manager?
Probably not if you do the data scraping yourself. I only use Manager from time to time now, mainly for viewing system Events (such as dropouts due to out of range line voltage and/or frequency or other notable events).
These discussions are very interesting,but I just wanted to point out that for anyone wanting a device to continuously and accurately monitor surplus production and divert it to heat water, you can’t do better than to purchase the Solar iBoost+. Simply clip a sensor on a wire in your meter box and it relays information to a box you can locate near your immersion heater. Even better, the device has a second output so that when water is hot, you can divert elsewhere – mine goes to a panel heater. Device then keeps checking periodically for water cooling due to circulation or use and flips back to water heating when required. I have also reduced the thermostat on my gas water heating circuit and maxxed it on the immersion thermostat. For about 7 months this year, I used no gas at all. Everything solar.
I’d like to have envoy type monitoring primarily to be sure none of my inverters have failed but don’t really want to buy an envoy. Has anyone been able to communicate directly with the inverters?
Long story shorter…Love reading this thread. I am ultimately working on an Excel sheet to pull historical data from Enphase API but after finding this thread, I am interested pulling realtime Envoy data to post on my Excel sheet for current production and consumption data. However, for the life of me, I can seem to make the math work. And here is what I am doing to compare what I am getting from production.json withe my envoy.local/home page. I am trying to take my production and consumption numbers at the same point and time as running production.json and attempting to match up. Never matching…I am assuming that the local home page is grabbing every minute so I am waiting to pull(json) right when it changes. Am I missing something in my retrieving of production and consumption via JSON? My json values are always lower than the local home page. Makes wonder how accurate I can get if I missing something. Can someone explain what I am seeing/experiencing? Or maybe a better way?
I’m having trouble getting the installer password. I tried to run the .apk on a couple devices, and an emulator but none were working. Enphase support was also entirely unhelpful in this matter. Can someone who has gotten it working provide the device and OS information?
What happens when you try to use / install the app?
Anyone knows how to determine the mapping of inverters to array (as graphically represented in the enphase web site or app)? I have 2 arrays and I’d like to be able to know how to group each individual inverter to sum production for each array as they are connected to different houses.
Hi thecomputerperson,
Just wondering if you can give me some hints on how to go about getting very basic info (which can be found from /production.json) and display it on a re-purposed iPhone. I want it to be super minimalist, so ideally a HTML file directly on the iPhone. But I do have a Raspberry Pi2 at my disposal I could setup to be a server. Any hints what would be the best approach? I’m not really confident in writing java script. But I guess I could learn if push comes to shove. Just want a basic dashboard in my kitchen, with near real time consumption, production, and net value.
Hi @Capt. Smek – I asked my installer and they were able to provide a picture of the layout and the matching serial numbers for each panel. Then I was able to replicate the Envoy display.
Hi @Oliver, some options to consider:
– I guess the Envoy mobile app doesn;t meet your needs?, I have Android but I assume there is an Apple version
– Look for existing integrations to home automation systems like Home Assistant
(Hopefully someone else has done the hard work :)
– On the Pi2 run a Cron job calling CURL to preiodically download the JSON data, then host an html page with javascript to render it
– On the Pi2 run a Python web app, for example Python using Flask to serve up the data, then render with javascript and html
– On the Pi2 run an app (like the Python examples) to query and store the data to InfluxDb, then use Grafana to viualise it
I do the last one using containers for Grafana, InfluxDB, and my python app. Grafana is great and pretty easy to use, and handles rendering on a mobile.
I expect various people have done the options above, and there is code and links in these comments that may help.
I plan to put my solution on GitHub with the container detail so it can just be installed, I’ll see how I go this weekend.
Hi all, I’ve added my container setup that reads from the Envoy and writes to InfluxDb here: https://github.com/jinxo13/EnvoyReader
Thanks for this post! I’m building a raspberry pi powered LED light strip power meter to visualize the our current solar production and total power consumption. I was going to use the enlighten API (I thought it was great that was documented as well as it was) but then figured there must be some way to grab that data off the local device. I found the webpage for the device but wondered how I could access it programatically… production.json!
I was playing with my Envoy-M (pill shaped) now that I have my installer password – thanks for that – and any of the /ivp/ urls redirected to port 9094. Authenticating with that and going to the root of the site gives me a directory listing with interesting pages, including
http://envoy:9094/ivp/api.html
which leads to full documention!
Good to know! Any chance you can pastebin, or similar, a copy?
Well I’ve been reading almost everything… you guys did a great job.
I’ve found my password from your app and I was playing around with the various urls.
I’m not a developper really, can read inside some code but not write.
I was wondering if someone already wrote some kind of program that will use the */stream/meter url ?
If I’m right, this in the only place where you can get the counter’s values every second without making the Envoy mad ?
That would be great to display real-time values or let’s say 1hour historical on tablet.
I thaught that I would have this kind of app when I purchased the Envoy-s Metered (https://guide.openenergymonitor.org/applications/solar-pv/) but it looks like Enphase is not willing to let us play around with real-time stats.
I only have normal non 3-phase and poll /production.json without much problem every 3 seconds but I do have to “back off” if it fails to respond. I’ve got that interval set to 30 seconds until it then gets a successful response again. As I don’t have 3-phase I’ve not bothered looking into the /stream/meter url.
My code is asp classic and not really elegant or of much use. I’m sure I’ve seen some forum posts with code in for OpenHAB and some other energy monitoring tools that reference my page here.
Yes I’ve seen some guys making programs using /production.json on github. I’ll try one for sure. The thing is that with the real time I was thinking that maybe we could divert the excess somewhere without having to get another box with CT.
What I would like to achieve in time is to use only the Envoy to tell an EV Wallbox to adapt it’s charging based on the energy produced in real time.
I don’t think I’ve seen something like that made, using only the data from the Envoy (S-Metered).
Anyway, great job for the app…
I use /production.json to do exactly that – dump export above 1.5kW into an electric heater in an outside building.
sCurrentProduction = findValue(productionSet, “wNow”)
in the production set from that file.
sCurrentConsumption = findValue(consumptionSet, “wNow”)
in the consumption set from that value
Work out if production is 1500w higher than consumption and turn on heater.
Thanks for the article
Is there a way to change the reporting mode from “Lo Bandwidth” to “Standard” without contacting the installer?
I already have the installer password from the APK and able to connecting the Envoy via Wifi.
Thanks
This can be done in the Enphase installer app under Network in the “Monitoring” box, click the little arrow down and you might have a blue “Switch to standard reporting” button or similar.
@thecomputerperson
Apparently, the installer kit app requires the installer-level credential in which my home-owner credential does not work. I thought there is way to update this reporting mode using a browser and “Installer login” credential
Maybe – what if you visit the envoyIP/installer in your browser and try to log in :D ?
This is so good! Thank you for sharing all this. I’ve just got myself hooked up with a proper data feed now and the per-panel graphs I get are so much better than what enlighten can offer.
Responding to Bat back in June last year… Using the information found on this and related pages, I put together some code to integrate the stream/meter event stream and feed it into my EmonCMS.
The details (and code) are all posted in the openenergymonitor forum here: https://community.openenergymonitor.org/t/another-enphase-envoy-s-integration-into-a-local-emoncms.
Very pleased I found this post, and your app!
Anyone figure out how to get Battery (Encharge) data direct from Envoy? T3
Reading the live stream data, I get negative values for production data for the three phases. Is this because I have not installed any CTs? Will it correct itself after I do install these?
Thanks for all the posts and knowledge with the Envoy.
Has any one had any knowledge with Envoy Failure.
Flash memory filled over the threshold limit, 94 %
This unit has never been connected a server. This is the second one to fail.
is there any way to delete the flash memory.
I have connected a USB to TTL UART and get the Linux boot
U-Boot 2013.10-rc3 (May 05 2015 – 11:22:14)
AM35XX-GP ES2.0, CPU-OPP2, L3-165MHz, Max CPU Clock 600 Mhz
ENVOYR Board + LPDDR/NAND
I2C: ready
DRAM: 256 MiB
U-Boot Enphase Part Number: 590-00015-r01-v01.01.02
Last Reset: POWER
Envoy Compat 3
Any help would be appreciated Thank you.
Larry
I have access to the API on Enphase, which has both production and consumption monitors. I have been pulling the data down every 5 minutes into my SQL Server. I can more or less figure out the net consumption, but need to understand how to compute what I am exporting to the grid and importing from the grid. I see these metrics in the Enlighten app, so am sure that the data is there. Any help is appreciated
@Sashi N – There are several projects on GitHub that poll the Envoy directly so you don’t have to depend on Enphase/Enlighten. Two off the top of my head – Enphase Collector, Solar Observatory. The source may help with the calculations. (Note: Searching on Enphase in GitHub helps. Searching on Envoy is problematic because there are other things with that name.)
I should also mention that some are using the home automation platform Home Assistant with the Envoy Reader integration. Out of the box it isn’t doing any of the calcs, but you can see in the community forum how some people have set up their dashboards. See the thread “Need testers for updated Enphase Envoy sensor” in the home assistant community.
@Rob Terzi, thanks! Using the code in Github, I think I have cracked this nut. It does not tie out to what I see in the Enlighten app, but is close enough. I am almost positive that it might be due to the power factor.
Hi, anyone is aware of some example coding in C++ for reading the stream data from the envoy ? I would like to readit on a small microcontroller like Arduino of ESP32
Anyone figured out how to get Encharge Battery data directly from Envoy?
if you just need to know battery percentage you can use this url
http://envoy/ivp/ensemble/inventory
each of your batteries will show up under ENCHARGE devices , in this json formatted data,
[
{
“type”: “ENCHARGE”,
“devices”: [
{
…
“percentFull”: 70,
…
}
]
},
{
“type”: “ENPOWER”,
“devices”: [
{
…
}
]
},
{
“grid_profile_name”: “…”,
“id”: “…”,
…
}
]
their may be other url i haven’t had the chance to try to find battery charge/discharge info.
first you will need to go to
http://envoy/info.xml
in the top device->software you can get your version, mine was D6.0.97
then i went to this url, put my software version without the ->D
this will output about 4689 lines of javascript
most of the javascript functions of interest are on line 3178-3182
http://envoy/backbone/application.js?version=06.0.97
Thanks, that’s very useful. Interesting that the first time I tried it I couldn’t get logged in.
Poked around some other screens and now the browser seems to know the right passwd and data comes out just like I was looking for.
T3
I played with it more, it needs the installer password. it also has a field for discharge/charge in Watts, the attribute name is “real_power_w”. Besides “percentFull” attribute, it has “temperature” for batteries and the smartswitch too.
These 2 attributes “Enpwr_grid_mode”, “Enchg_grid_mode” seem to describe whether i’m off grid. right now both of them have the value “multimode-ongrid”
the output of the URL is definitely JSON data structure, i played with custom rest sensor in home assistant to get some of this data parsed and show up on a dashboard in home assistant. I’m hoping to setup some automation to trigger turning off smart plugs in the house if reserve battery power goes below 30% during power outage, and if my internet is still working, to turn off HVAC (nest thermostat) automatically.
Do you have any python code that includes the passwd so you can get the data in a program?
I am struggling to make that work.
T3
No, even though home assistant is written in python, most of the setup is yaml. some things i shell to the raspbian to execute in linux bash shell. home assistant already has integrations for REST url calls like this.
here’s how you could run a curl command to retrieve the data.
curl –digest –user installer:PASSWORD -X POST -H “Content-Type: application/json” http://envoy/ivp/ensemble/inventory
if python can shell to linux bash, maybe you could pass in this curl command, and capture its output.
Thanks! I’ll post if I get something working.
T3
Thanks to contributors in this thread, I’ve modified a few scripts and have created a Python script that takes the stream/meter information and feeds it to a mqtt broker. I then use it for real-time power display in Home Assistant. https://github.com/vk2him/Enphase-Envoy-mqtt-json
Thanks for all your work on this. Here is a link to a Grafana/Postgresql screenshot of my setup. I log 1 minute extracts to postgres and present with grafana
https://imgur.com/gallery/TYSs0mb
Ian Mills, thanks for doing this. Just a word of caution. .You have given your passwords in the code. I would recommend masking any sensitive information
Oeps, first posted my question in the wrong topic.
I’ve been using the script for quite some time now. It works great. A huge thank you to everyone who contributed.
Now I have a little problem. Domoticz has been out for a day and a half. Now the production data is no longer correct. Is there a possibility to correct the production figures for the time when Domoticz was down?
Hi, is it possible to see the data of the microinverters work performance? thanks
For those interested, I’ve used this password generator to create a Home Assistant addon that outputs mqtt from the json steam http://envoy.local/stream/meter
It’s easy to install and configure via a single click of the “add Repository” button in the instructions:
https://github.com/vk2him/Enphase-Envoy-mqtt-json
Hi, I stumbled on this forum while writing my own app to balance high demand appliances based on our private solar production on our home. It is a terrific find! Right now, I am using the local ‘api’ http://%5Bx.x.x.x%5D/production.json, and currently using “wNow”.
I am working on converting to the new cloud APIs Version 4. I have written the necessary Authentication code (it was kind of a nightmare, but it is done!).
Now I am trying to get current production (the equivalent to the “production.json.wNow”). However, using the V4 /summary API, I can only get “current_power” and its value is almost half of the “wNow” when requested at almost the same time. So I don’t trust it. It does not even match what is indicated in the Enphase app whereas the “wNow” matches the Enphase app.
Does anyone know what “current_power” is based on?
I think I found the data I was looking for. After calling the telemetry URL, the JSON payload contains an array of ‘intervals’ in descending order. Within the ‘interval[x]’ values, the current power is ‘.powr’. It is very close to the same number from the local api /production.json ‘wNow’.
So I check every 5 minutes, find the length of the returned interval array, select the last record (length -1) then pull the ‘powr’ value to get the current production level. Once I have this value, I can compare it to my threshold values set to determine if it should turn on or off a specific appliance (such as a swimming pool pump, car charger, HVAC, etc) via their APIs.
This is the cloud API URL: https://api.enphaseenergy.com/api/v4/systems/{systemid}/telemetry/production_micro?granularity=day.
let rec = msg.payload.intervals.length – 1;
let val = msg.payload.intervals[rec].powr;
Pingback: Get-EnphaseProduction.py | greiginsydney.com
Pingback: Get-EnphaseData.py | greiginsydney.com
Is anyone else seeing an issue where you now suddenly need to provide credentials to access the /production.json URL on their Envoy system? Since two days, the script I used to poll production data gets redirected from HTTP to HTTPS which uses a self-signed cert and needs logging in:
Trying 192.168.X.Y…
Requesting https://192.168.X.Y/production.json
ftp: Error retrieving https://192.168.X.Y/production.json: 401 Unauthorized
I’ve tried the ‘installer’ credentials, but get the same result.
So it looks like a firmware update basically disabled my local access to the data.
Same here. Need to go to https://entrez.enphaseenergy.com/login_main_page and write Enlighten user/password, etc to login.
Will ask support to downgrade my Envoy-s firmware. :-)
Yes, my scripts to download production data started failing on June 7, 2023. Has anyone figured out how to locally access these pages?
I don’t mind providing a password, but of course I want the authentication done locally, not involving Enphase. If they go bankrupt, are sold, or are hacked, the system would become a brick. Not to mention that it really isn’t their business what I am doing with my data on my system.