Python script not processing files

[Docfxit]

I'm trying to run a python script on a secure folder. The script is running. I don't think (although I don't know for sure) that the script is reading the files.

Running as Administrator
The folder permissions and all sub folders/files is:
Administrator (Full Control)
Owner Administrator

Running in Win7 sp1

Batch file:

cd "C:\Programs\CommuniGate Files\SystemLogs"
Call "C:\Batch\CGPLogSummary.py" 
cmd

The python script is supposed to read files, Process the files, and send an email with the results.
The script does send the email but the results are not in the email.

Does anyone know what I can do to figure out what is wrong?

Thanks,

Docfxit

 

[MilesAhead]

Could you post the Python script?

 

[Docfxit]

The first part is at:
pastebin.ws 2.1 - Reading paste 7ipf3
The second part is at:
pastebin.ws 2.1 - Reading paste 2fuu3n

It was too large to post in one pastebin.

I am running it in Python 3.2.2
Win7

Thank you for taking a look at it.

Docfxit

 

[MilesAhead]

And when it runs you are on a remote machine so you can't see any error messages?

On Windows whenever I see file paths with spaces, like somevar = "C:\Program Files\yada yada\some.exe"

I look for an error due to the file path being chopped off at the first space like "C:\Program is not a valid path" since spaces are used as a separator on the command line. Worst thing that ever happened to Windows. I bet it accounts for more than 80% of programming bugs with files.

 

[Docfxit]

And when it runs you are on a remote machine so you can't see any error messages?

On Windows whenever I see file paths with spaces, like somevar = "C:\Program Files\yada yada\some.exe"

I look for an error due to the file path being chopped off at the first space like "C:\Program is not a valid path" since spaces are used as a separator on the command line. Worst thing that ever happened to Windows. I bet it accounts for more than 80% of programming bugs with files.

Thanks for looking at it.

I am on the local machine when this is run.

I'm not seeing any errors when it's run. I'm running it from a bat file.
Is there something I can put into the script to test the success or failure of the path access?

Thank you,

Docfxit

 

[Pyprohly]

Hi, excuse my abrupt entrance to this thread,

When asking for help on a script, it's probably a good idea to post the script or at least provide snippets of it. It's difficult to help people with problems such as "My script isn't working, I don't know what the problem is, please help me".

If you expect decent help you must be specific about your problem... Are there any error messages? Or does the script simply hang (no output whatsoever)? You should make some attempt to find out (more information about) the problem prior to resorting to a help forum. You should post any findings you feel are necessary to help others assist you in your problem.


With that out of the way, Docfxit, I have recovered your Python script from the Pastebin links you've given and have reposted your script in a zipped file (attached to this post) for the sole convenience of others, should they be able to help you.

I've examined the Python script, and just as MilesAhead suspected, there are immediate errors with your paths (but hardly to do with spaces); they are entered incorrectly in a Python script nonetheless. I assume you've entered these values yourself, Docfxit.

logdir = "C:\Programs\CommuniGate Files\SystemLogs\"
submitdir = "C:\Programs\CommuniGate Files\Submitted\"

Two things wrong with the above here.

First of all, specifying a folder path with a trailing backslash is considered incorrect. And this is anywhere in Windows. Try not to add an excess backslash to the end of paths, ever.

Secondly, in Python strings, a backslash is an escape character (a.k.a EOL character). If you want to use a literal backslash, you must escape it by backslash-ing the backslash.
E.g. logdir = "C:\\Programs\\CommuniGate Files\\SystemLogs"

Alternately, you could specify to Python a raw string (or literal string) where each character in the string is interpreted literally. Do this by appending an 'r' to the start of a string.
E.g. logdir = r"C:\Programs\CommuniGate Files\SystemLogs"


Lastly, there is a very minor syntax error on line 904, where

outstring += (val + ":").ljust(stats_len + 2) +

should have an EOL (end of line) character on the end of the line. For instance:

outstring += (val + ":").ljust(stats_len + 2) + \

The EOL character is need on the end here because this line of Python code continues on the next line. I have no idea why it is missing one.


It is up to you, Doc., to find and fix these three bad lines (hint: 10, 11, 904).


On a side note, your Batch file, you should be using the 'pause' command instead of 'cmd' if you want to view the scripts' output. (pause halts script execution until a keypress.)

 

[MilesAhead]

It's been a very long time since I played with Python. Pyprohly points out that the backslash is an escape similar to C based languages.

A good idea is to make tiny test scripts to find out if a particular thing is working as expected. For instance assigning the path to a variable then prining it out to the console to see the output.

There are many Python programmers on Windows so it may be a good idea to post questions in a Python forum. Instead of guys like me who have forgotten most of their Python you would have your question read by Python programmers current in the language. I think the last time I wrote a script Python was in version 2.x

 

[Docfxit]

Thank you very much for jumping in and explaining everything in detail for me.

Hi, excuse my abrupt entrance to this thread,

When asking for help on a script, it's probably a good idea to post the script or at least provide snippets of it. It's difficult to help people with problems such as "My script isn't working, I don't know what the problem is, please help me".

I'm sorry I didn't include enough information.

If you expect decent help you must be specific about your problem... Are there any error messages? Or does the script simply hang (no output whatsoever)? You should make some attempt to find out (more information about) the problem prior to resorting to a help forum. You should post any findings you feel are necessary to help others assist you in your problem.

I will try to give more information next time.

With that out of the way, Docfxit, I have recovered your Python script from the Pastebin links you've given and have reposted your script in a zipped file (attached to this post) for the sole convenience of others, should they be able to help you.

Thank you for including it in this post correctly.

I've examined the Python script, and just as MilesAhead suspected, there are immediate errors with your paths (but hardly to do with spaces); they are entered incorrectly in a Python script nonetheless. I assume you've entered these values yourself, Docfxit.

logdir = "C:\Programs\CommuniGate Files\SystemLogs\"
submitdir = "C:\Programs\CommuniGate Files\Submitted\"

Two things wrong with the above here.

First of all, specifying a folder path with a trailing backslash is considered incorrect. And this is anywhere in Windows. Try not to add an excess backslash to the end of paths, ever.

I agree with you 100%. This is a script I acquired from someone else and I am not familiar with Python.
I have removed the trailing backslash from both strings. I am now getting an error when I run the script.
Error: [Errno 2] No such file or directory: 'C:\\Programs\\CommuniGate Files\\Sy
stemLogs2014-12-05.log'

There should be a backslash between the "s" and the "2". Since it is not a good practice to have a trailing backslash it would be great if you could please help me by suggesting how to code that backslash in the script.

Secondly, in Python strings, a backslash is an escape character (a.k.a EOL character). If you want to use a literal backslash, you must escape it by backslash-ing the backslash.
E.g. logdir = "C:\\Programs\\CommuniGate Files\\SystemLogs"

In the original script it did have a double backslash. For some reason it must have been dropped when I posted it in Pastebin.

Alternately, you could specify to Python a raw string (or literal string) where each character in the string is interpreted literally. Do this by appending an 'r' to the start of a string.
E.g. logdir = r"C:\Programs\CommuniGate Files\SystemLogs"


Lastly, there is a very minor syntax error on line 904, where

outstring += (val + ":").ljust(stats_len + 2) +

should have an EOL (end of line) character on the end of the line. For instance:

outstring += (val + ":").ljust(stats_len + 2) + \

The EOL character is need on the end here because this line of Python code continues on the next line. I have no idea why it is missing one.

In the original script it did have a backslash. For some reason it must have been dropped when I posted it in Pastebin.

It is up to you, Doc., to find and fix these three bad lines (hint: 10, 11, 904).


On a side note, your Batch file, you should be using the 'pause' command instead of 'cmd' if you want to view the scripts' output. (pause halts script execution until a keypress.)

Thank you very much. That is great of you to help so much.

If you could let me know how/where to code the slash between the path and the file name that would be wonderful.

Thanks a bunch,

Docfxit

 

[MilesAhead]

I should have posted a link to get you started:
https://www.python.org/about/help/

 

[Docfxit]

Thank you for the link.

For the temporary moment I added the backslash backslash back in to get it running.

I ran the script. I am getting the email that the script sends with the dates. It's not including the data in the email that it's supposed to send. I am not getting any error messages.

Thanks,

Docfxit

 

[Pyprohly]

I should have posted a link to get you started:
[URL]https://www.python.org/about/help/[/URL]

If you ever plan to program, I do highly recommend Python.

Error: [Errno 2] No such file or directory: 'C:\\Programs\\CommuniGate Files\\Sy
stemLogs2014-12-05.log'

Error message; excellant.
I did actually get the same error myself. I was waiting for you to replicate this error, so I know you're on track.

There should be a backslash between the "s" and the "2". Since it is not a good practice to have a trailing backslash it would be great if you could please help me by suggesting how to code that backslash in the script.

Correct, there should be a backslash between those characters and it is not your fault that is doesn't. It's the programmers.

In line 1170, paths are joined (and quite terribly too, I will add). It lazily squashes two strings together when it should really join the two strings as a path. This is why backslashes are missing. A very sloppy mistake if you ask me.

fnamewithpath = logdir + fname

Your mission: locate the above line (at 1170) and replace it with the following:

fnamewithpath = join( logdir, fname )

(Note: This join function is an attribute of os.path, which was imported at the beginning of the script with: from os.path import *)
(Important Note: Indentation is compulsory/necessary in Python. So make sure any whitespace to the left of a line is preserved or you will have problems)


Also, about your batch file again, the cd command is very optional. There is no use in it if all the paths you are working with are already written in the Python script.

 

[Docfxit]

That is really really nice of you to help me get the coding correct.

I feel much better now that it follows the standard way of supplying a path.

I have applied your suggestion and removed the double backslash at the end of the path.

It runs without errors.

Now If I only knew why it isn't outputting data in the email correctly it would be great.

I have uploaded a test file just in case you would like to take it for a spin.

Thank you,

Docfxit

PS: Currently the test file below needs to be in a folder called:
logdir = "C:\Programs\CommuniGate Files\SystemLogs"

Of course you could change that line in the script to any folder you like.

 

[Pyprohly]

I'd be happy to take the script 'for a spin' for you, but with backslashes missing all over the version I have of your script, I can't be sure if the output I'm getting is right.

Please remove any personal information-al inclusions you've made to your script and repost the Python script here in a zip file, It'll help me better debug the script for you.


Also, if you would like to lightly explain what the script is actually meant to do, it'll save myself a bit of reading.


Meanwhile, another path related problem resides on line 1191, similar a problem to which line 1170 had with joining paths.
The corrected code for 1191 should be:

outfilename = join( submitdir, "ls" + str(time.time()) + "-" + str(os.getpid()) )

Assuming the output file name is supposed to look something like
ls1417926446.033319-7848.sub
.

 

[MilesAhead]

If you ever plan to program, I do highly recommend Python.

I've been programming, mostly as an avocation, since 1986. Lately I've been doing mostly hotkey utilities in AHK. I dabbled in Python for a bit in the past. I downloaded the ActivePython 3.x installer in case I decide to brush up.

The only C++ program I've written in a long time is MD5Hash. It's just too much work to do it for fun unless there's some reason to use it(such as a fast MD5Sum routine I found in that case.) Plus I had to try the Lamda Expressions just for grins.

 

[Docfxit]

I'd be happy to take the script 'for a spin' for you, but with backslashes missing all over the version I have of your script, I can't be sure if the output I'm getting is right.

I'd be happy to attach the zip file.

Please remove any personal information-al inclusions you've made to your script and repost the Python script here in a zip file, It'll help me better debug the script for you.


Also, if you would like to lightly explain what the script is actually meant to do, it'll save myself a bit of reading.

The script reads the log files from the day before the script runs. It summarizes what has happened in the log files. And sends an email with the summary. An example of what the summary looks like is in the file EmailExample.zip

Meanwhile, another path related problem resides on line 1191, similar a problem to which line 1170 had with joining paths.
The corrected code for 1191 should be:

outfilename = join( submitdir, "ls" + str(time.time()) + "-" + str(os.getpid()) )

Assuming the output file name is supposed to look something like
ls1417926446.033319-7848.sub
.

The input file is called logsummarytest.zip

Thank you very much for taking a look at it,

Docfxit

 

[Pyprohly]

And sends an email with the summary.

If you are under the impression that this Python script sends an Email, it attempts nothing of the sort.

The only output this script yields is a text file. Is this not the sort of output are you are after?


If the test log file you attached in post 12 is fed to the script as input, what should the output file look like? To me it looks nothing like what EmailExample.sum is, which I'm guessing it should.

It appears a function named "generateReport" (in the "CGPLogReporter" class) always returns an empty string which should instead (as the function name implies) contain the generated report summary.

I'll examine further and see if I can't find out why.

 

[Docfxit]

And sends an email with the summary.

If you are under the impression that this Python script sends an Email, it attempts nothing of the sort.

The only output this script yields is a text file. Is this not the sort of output are you are after?

After getting your post I looked into how this could be possible. I found that when a text file is put into the submitdir the server automatically sends it out as an email.

If the test log file you attached in post 12 is fed to the script as input, what should the output file look like? To me it looks nothing like what EmailExample.sum is, which I'm guessing it should.

I'm guessing what you are seeing is a detail of the log. I'm guessing (Maybe in "generateReport" the detail gets sorted and summarized.

It appears a function named "generateReport" (in the "CGPLogReporter" class) always returns an empty string which should instead (as the function name implies) contain the generated report summary.

I'm guessing that is where the problem is. I think once you know why it returns an empty string you will know why it doesn't produce the output text file.

I'll examine further and see if I can't find out why.

Thank you very much for looking at it.

Docfxit

 

[Docfxit]

I just remembered the script only processes log files from The day before it's run

So if it's run today 12/8/2014 it will process log files from 12/7/2014
I don't know if it uses the file name for the date or the date the file was created.

Docfxit

 

[Pyprohly]

I just remembered the script only processes log files from The day before it's run

I am aware of this and have had to rename the test log file each day accordingly -- the script checks the file names date rather than checking the file creation date which is odd and annoying.

Also, an argument can be passed to override the days offset. 1 (default) is yesterday, 2 is the day before yesterday, and so on.


I've found out why the generateReport function keeps returning an empty string.

Lines 831 - 837 read:

        stats = {}
        spammers = []
        unknowns = {}
        blacklists = {}
        spamtraps = {}
        contentRejections = {}
        relayRejections = {}

The above braces (dictionary data type) and brackets (a list data type) are all, as you can see, empty. It seems the genorate report function only acts upon the data that is supposed to be within these braces. But what stumps me is that there is absolutely nothing in these braces and there never will be. Thus the generateReport function always returns an empty string no matter what file the script reads!

I have not a single clue of how EmailExample was created using this script the way it is exactly.

Something should be in at least some of all these pairs of braces and brackets and I'm not sure what, but unless you fill at least some of these braces with god-knows-what, you're left with the only option of contacting the developer of this script and getting support from them instead.

 

[MilesAhead]

The above braces (dictionary data type) and brackets (a list data type) are all, as you can see, empty. It seems the genorate report function only acts upon the data that is supposed to be within these braces. But what stumps me is that there is absolutely nothing in these braces and there never will be. Thus the generateReport function always returns an empty string no matter what file the script reads!

It looks like the assignments are just placeholders to be filled in later. IOW equivalent to func myfunc { //implement myfunc here } in C++