Troubleshooting Manual

 


I'm having problems with the Splunk PDF Server app

I'm having problems with the Splunk PDF Server app

This topic is about troubleshooting the PDF Server for Linux app.

The PDF Report Server App allows Splunk to email saved searches and reports in a printable PDF format. Once installed, it allows any Splunk instance on the network to format saved searches and reports as PDF files by making requests to the instance of Splunk with the app installed.

For information about the integrated PDF generation feature introduced in Splunk Enterprise 5.0, see Generate PDFs of your reports and dashboards in the Reporting Manual.

Commonly experienced symptoms

I'm not getting any emails

PDF Server depends completely on the alert scheduler. Make sure that you can receive alert emails before proceeding to the next step of troubleshooting. "Set up alert actions" in the Alerting Manual can help you with this.

Remember to give the PDF Server app 3-5 minutes to put into effect any configuration changes you make.

I am receiving emails for some alerts but not my scheduled dashboards

Scheduled dashboards are scheduled by the system via auto-generated saved searches. Check that the cron schedule assigned to the dashboard is the interval that you expect (via Manager > User interface > View PDF scheduling).

There are also some included dashboards relating to the scheduler. You can find these from the Search app search screen or summary dashboard. Above the search bar, click on Status > Scheduler activity.

I am receiving emails for scheduled searches/dashboards with a PDF error message

A common error message that might be included in the body of an alert email is:

An error occurred while generating a PDF of this report:
Failed to generate PDF: Appserver failed to dispatch report request to
/services/pdfserver/renderpdf: 500 Internal Server Error

This message means that the PDF server was successfully called by the scheduler, but was unable to either start Firefox or capture any HTML output. The general steps to troubleshoot this are:

1. Verify that the PDF server is installed on a Linux system.

2. Inspect $SPLUNK_HOME/var/log/splunk/python.log.

Why are there a bunch of _ScheduleView_ items in my list of saved searches?

Splunkd has only one scheduling mechanism: the saved search alerter. Therefore, scheduled PDF delivery of views can be run only via this existing channel. Future versions of Splunk will separate the saved search and alerting concepts.

I'm getting a PDF generated, but the output is not what I expected

I have a wide table that has a lot of columns, but only the first N columns are showing

This bug (SPL-32328) was fixed in version 1.3 of the PDF Report Server app. If you can't get the new version for some reason, try setting the page setup to landscape mode (in Settings > Email settings), instead of portrait mode, to give the browser some more room to render information.

My pages are printing, but everything seems to be squished together/rendered very small

Your dashboard might have too many columns.

While it might seem logical that whatever is shown on the screen can be reproduced on paper, the reality is that the canvas and DPI of an infinitely scrollable monitor is completely different than the canvas and DPI of a physically constrained piece of paper. As a result, when Firefox prints a page, it first imposes real-world constraints on the web page.

Most notably, Firefox shrinks the page width to about 700 pixels. For reference, a typical browser instance running on a 15" laptop screen is about 1300 pixels. Therefore, pages that look good on a screen are significantly different once it washes through the browser's print engine. A dashboard with three charts in a row might look good on screen, but when made to fit into 700 pixels, will probably leave much to be desired.

Most people don't realize just how severe the constraint is of a typical printed page. See this demo on the Community Wiki to get a sense of how small your browser window needs to be to reflect the needs of a printed page.

Possible remedies are:

  1. Create an alternate dashboard, dedicated for printing, that lays everything in one column.
  2. Use custom CSS and charting properties to strip unwanted elements from the page.
  3. Remove panels that don't need to be printed, or split them into separate dashboards.

My PDF is being generated but the table or timeline doesn't seem to have any data points

This is most commonly caused by trying to schedule a saved search that was created outside of Splunk Web. By default, Splunk attempts to dispatch searches as fast as possible, which means that it does not enable features like events, timeline, or field summaries -- things that the main UI needs to render the timeline. Therefore, sending a scheduled job through the PDF server will only yield whatever the dispatch system retained.

Possible remedies are:

  1. Only schedule searches created through the report builder.
  2. Manually edit the savedsearches.conf file to have "dispatch.status_buckets=300" set.

In general, changing these settings will slow down a scheduled search. You'll want to understand the relationship of status_buckets (in limits.conf) and required_field_list to search dispatching in order to make an informed decision about what you want.

My dashboard is printing, but some panels still say "loading" or "waiting for results"

The most common problem is a dashboard that does not use any scheduled searches, instead relying on ad-hoc searches that take a long time to complete. By default, the PDF server will wait up to 3 minutes for page to finish loading all of the data. If individual searches fail to complete within that time, that's when the "waiting" message will show.

Possible remedies are:

  1. Avoid ad-hoc searches. Schedule searches instead.
  2. Increase the firefox_timeout setting in pdf_server.conf.

An error message related to "connection" is returned

Either a PDF is returned with just an error message or the alert email contains an error relating to connection problems. This is most likely related to network routing or SSL-related issues. Contact Splunk Support.

The fonts are all messed up in the PDF

Make sure you have installed Dejavu fonts on your Linux system. The PDF Server app needs the OS to have basic fonts installed, like the bitstream-vera-fonts package.

Version 1.1 of the PDF Server app resolved many font issues. If you've checked that Dejavu fonts are installed and you're still seeing a font problem, contact Splunk Support.

Charts are split over a page break

Sometimes a chart should appear on one page, but it's broken in the middle and split across two pages. This is a known issue. There's no good workaround right now other than not having too many charts on a page.

My PDF is truncated after the first page

Sometimes a PDF that should be several pages long has only one page, or only the first few pages. This was improved in version 1.3 of the PDF Report Server app. If you're using the latest version and still seeing this problem, try limiting the number of panels you're printing to PDF if you can.

I upgraded from 4.2.x to 4.3 and then PDF Report Server stopped working

Check your server's python.log for a message like this:

2012-02-19 07:01:00,444 ERROR An error occurred while generating a PDF of this report: Failed to generate PDF: Appserver failed to dispatch report request to /services/pdfserver/renderpdf: Splunkd daemon is not responding: ('The read operation timed out',)

If you are seeing this message after upgrading to 4.3, read through this Answers post to diagnose and work around the issue.

How does this thing work?

The PDF Report Server App uses Firefox (which is shipped with the app) to render the reports built from Splunk. Whether it's a local or remote installation, the basic flow is:

  • A Splunk alert is triggered, starting the PDF reporting process (visiting the status page will also start the process). A script named sendemail.py makes a request of splunkweb.
  • The local splunkweb makes a call to the splunkweb acting as the PDF server and waits for a response from the PDF server.
  • The PDF server triggers local Python, which starts Firefox and X.
  • Firefox contacts the local splunkweb to trigger the search.
  • The Splunk server sends search results (once completed) to the PDF server for rendering.
  • The PDF server sends the rendered PDF back to the Splunk server.
  • The response is then included in an email alert.

This flow takes a few minutes to complete. Keep that time lag in mind: be patient when you've changed a configuration and are waiting for your change to manifest.

What does the logging output of PDF server in python.log look like?

Here's a python.log entry for a successful print job:

  2010-02-22 23:52:26,295 INFO    xvfb:45 - Started new X server. Now 1 active
  2010-02-22 23:52:26,295 INFO Started new X server. Now 1 active
  2010-02-22 23:52:26,295 INFO    pdfhandler:317 - Executing
['/home/johndoe/corpdev/opt/current/etc/apps/pdfserver/bin/firefox-x86_64/firefox',
'-print', 'http://purpleboo:2600/app/search/dashboard', '-printmode',
'pdf', '-printfile', '/tmp/tmpBRkk64', '-title', 'Splunk Alert: 
_ScheduledView__dashboard', '-orientation', 'portrait', '-paperwidth',
'216', '-paperheight', '279', '-mode', 'splunk', '-cookie',
'session_id_2600=95dc64472d303d62f018b030a05ae17985526a09']
  2010-02-22 23:52:26,295 INFO Executing 
['/home/johndoe/corpdev/opt/current/etc/apps/pdfserver/bin/firefox-x86_6 
/firefox', '-print', 'http://purpleboo:2600/app/search/dashboard',
'-printmode', 'pdf', '-printfile', '/tmp/tmpBRkk64', '-title',
'Splunk Alert: _ScheduledView__dashboard', '-orientation', 'portrait',
'-paperwidth', '216', '-paperheight', '279', '-mode', 'splunk', '-cookie',
'session_id_2600=95dc64472d303d62f018b030a05ae17985526a09']
  2010-02-22 23:52:26,295 INFO    xvfb:194 - Assigned DISPLAY :5
  2010-02-22 23:52:26,295 INFO Assigned DISPLAY :5
  2010-02-22 23:52:26,305 INFO    xvfb:114 - Starting X Server: ['/usr/bin/Xvfb', ':5', '-screen', '0', '1024x768x24']
  2010-02-22 23:52:26,305 INFO Starting X Server: ['/usr/bin/Xvfb', ':5', '-screen', '0', '1024x768x24']
  2010-02-22 23:52:26,306 INFO    xvfb:115 - Starting X Server env: {'XAUTHORITY': '/home/johndoe/corpdev/opt/current/var/run/splunk/xvfb/xauth-5/Xauthority'}
  2010-02-22 23:52:26,306 INFO Starting X Server env: {'XAUTHORITY': '/home/johndoe/corpdev/opt/current/var/run/splunk/xvfb/xauth-5/Xauthority'}
  2010-02-22 23:52:26,310 INFO    xvfb:117 - X Started
  2010-02-22 23:52:26,310 INFO X Started
  2010-02-22 23:52:32,765 INFO    pdfhandler:338 - Generated pdf file.  bytes=29782 time=6.47
  2010-02-22 23:52:32,765 INFO Generated pdf file.  bytes=29782 time=6.47
  2010-02-22 23:52:32,772 DEBUG   pdfhandler:415 - statuslog: ['startup', 'sendprintstart', 'delayedshow', 'settings', 'print', 'success']
  2010-02-22 23:52:32,772 DEBUG statuslog: ['startup', 'sendprintstart', 'delayedshow', 'settings', 'print', 'success']
  2010-02-22 23:52:32,772 WARNING xvfb:127 - Stopping X Server 5
  2010-02-22 23:52:32,772 WARNING Stopping X Server 5


And here's an entry for an unsuccessful job:

 
  2010-02-23 11:10:08,184 INFO    xvfb:45 - Started new X server. Now 1 active
  2010-02-23 11:10:08,184 INFO Started new X server. Now 1 active
  2010-02-23 11:10:08,184 INFO    pdfhandler:317 - Executing 
['/home/johndoe/corpdev/opt/current/etc/apps/pdfserver/bin/firefox-x86_64/firefox',
'-print', 'http://purpleboo:2600/app/search/dashboard', '-printmode', 'pdf',
'-printfile', '/tmp/tmpJQ5v2N', '-title', 'Splunk Alert:
_ScheduledView__dashboard', '-orientation', 'portrait', '-paperwidth', '216',
'-paperheight', '279', '-mode', 'splunk', '-cookie', 
'session_id_2600=97e51957138653f0c79c6fc226cbc7479d849a60']
  2010-02-23 11:10:08,184 INFO Executing 
['/home/johndoe/corpdev/opt/current/etc/apps/pdfserver/bin/firefox-x86_64/firefox', 
'-print', 'http://purpleboo:2600/app/search/dashboard', '-printmode', 'pdf',
'-printfile', '/tmp/tmpJQ5v2N', '-title', 'Splunk Alert:
_ScheduledView__dashboard', '-orientation', 'portrait', '-paperwidth', '216', 
'-paperheight', '279', '-mode', 'splunk', '-cookie',
'session_id_2600=97e51957138653f0c79c6fc226cbc7479d849a60']
  2010-02-23 11:10:08,185 INFO    xvfb:194 - Assigned DISPLAY :5
  2010-02-23 11:10:08,185 INFO Assigned DISPLAY :5
  2010-02-23 11:10:08,194 INFO    xvfb:114 - Starting X Server: ['/usr/bin/Xvfb', ':5', '-screen', '0', '1024x768x24']
  2010-02-23 11:10:08,194 INFO Starting X Server: ['/usr/bin/Xvfb', ':5', '-screen', '0', '1024x768x24']
  2010-02-23 11:10:08,195 INFO    xvfb:115 - Starting X Server env: {'XAUTHORITY': '/home/johndoe/corpdev/opt/current/var/run/splunk/xvfb/xauth-5/Xauthority'}
  2010-02-23 11:10:08,195 INFO Starting X Server env: {'XAUTHORITY': '/home/johndoe/corpdev/opt/current/var/run/splunk/xvfb/xauth-5/Xauthority'}
  2010-02-23 11:10:08,199 INFO    xvfb:117 - X Started
  2010-02-23 11:10:08,199 INFO X Started
  2010-02-23 11:10:08,206 WARNING pdfhandler:323 - Failed to start Firefox for x86_64: [Errno 13] Permission denied
  2010-02-23 11:10:08,206 WARNING Failed to start Firefox for x86_64: [Errno 13] Permission denied
  2010-02-23 11:10:08,207 ERROR   pdfhandler:336 - Execution of Firefox for x86_64 failed
  2010-02-23 11:10:08,207 ERROR Execution of Firefox for x86_64 failed


In some cases you might get more information by enabling web debugging. Read more about web debugging in "Enable debug logging" in this manual.

What does a successful PDF job look like in scheduler.log?

There is no difference in logging event output other than the expected variations like name, user, time, sid:

  02-22-2010 23:23:04.687 INFO  SavedSplunker - savedsearch_id="admin;search;_ScheduledView__dashboard", user="admin",
app="search", savedsearch_name="_ScheduledView__dashboard", status=success, scheduled_time=1266909780, 
dispatch_time=1266909781, run_time=0.012, result_count=0, alert_actions="email",
sid="scheduler__admin__search_X1NjaGVkdWxlZFZpZXdfX2Rhc2hib2FyZA_at_1266909780_1778446329", thread_id="AlertNotifierWorker-1"

Troubleshooting steps

  1. Is PDF Server installed correctly?
    • Did you meet the system requirements?
    • You must unpack the PDF Server app in the $SPLUNK_HOME/etc/apps directory.
    • Is the xauth package installed correctly on the PDF server? Type "which xauth" on your command line. Your system should return "/usr/bin/xauth".
    • Is the Xvfb (X virtual framebuffer) package installed on the PDF server? Type "which Xvfb" on your command line. Your system should return "/usr/bin/Xvfb". Xvfb is not shipped as part of the app.
    • Check the settings in pdf_server.conf on the PDF server to make sure the above packages match the correct directory location.
    • The PDF Server app needs to be enabled in the system settings page in the UI. Check alert_actions.conf.
  2. Are you able to display a test PDF with the status page?
  3. Navigate to the status page. Within about a minute, a PDF should be sent to your browser. Be sure to read the message written on the PDF. If yours has an error message, check through the PDF Server setup instructions in the Installation Manual.
    [settings]
    screenshot_enabled = True
    restrict_to_splunkweb = False
    
  4. Does the search/report take more than a minute to return results?
    • Since the PDF server waits for the Splunk process to send search results, a long-running search might not get sent to the PDF server before it times out. If this is happening, edit pdf_server.conf on the PDF server. Change the Firefox timeout setting to be larger than the default of 180 seconds. Something like 300-400 seconds should do it.
    [settings]
    firefox_timeout = 400
    

    Additional specific tips

    • Note that on an x86_64 install, Splunk will try to run the x86_64 version of PDF Server, and then if that fails, it will try the i386 version. This means that in the case of a problem with the x86_64 version you will always see messages about i386 as well. On i386 Splunk, only the i386 version is attempted.
    • To schedule PDF alerts, you must be able to schedule searches in general. With Splunk's default settings, this means you must be at least a power user. Read about adding and editing roles in Securing Splunk.
    • The hostname parameter in the alert_actions.conf email stanza dictates the address used for calling the app server. This host/address must resolve on the local/remote system. If it doesn't resolve, you might see a "connection refused" error. If you specify a host name in the email settings configuration, alert_actions.conf, or in a specific alert in action.email.hostname, the host name must resolve on both the local Splunk node, and on the PDF server. If you do not specify a host name, the host name returned by the local operating system must similarly resolve.

This documentation applies to the following versions of Splunk: 6.0 , 6.0.1 , 6.0.2 , 6.0.3 , 6.0.4 , 6.0.5 , 6.0.6 , 6.1 , 6.1.1 , 6.1.2 , 6.1.3 , 6.1.4 , 6.2.0 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!