AppDynamics upgrade / Glassfish Master password / Keystore issues?

We were having real problems upgrading AppDynamics from 4.2 to 4.3 then on to 4.5. All of the issues where caused by things that were not mentioned in their documentation.

Hopefully this information might prove to be useful to someone else in the same situation.

Our main issue was been caused because we had a custom password securing our Glassfish Master / Keystore & Keystore Keys, all of which was the same, as they need to be, because of this the installer was failing as it doesn’t ask you to insert your password as a parameter during installation, you need to change it back to the default of ‘changeit‘ before commencing an upgrade when we tried to get asadmin to do this for us, we got the error:

asadmin change-master-password --savemasterpassword=true
Enter the current master password>
Enter the new master password>
Enter the new master password again>
Keystore was tampered with, or password was incorrect
Command change-master-password failed.

After many hours we worked out that you first needed to:
Browse to the folder: \AppDynamics\Controller\appserver\glassfish\domains\domain1\config and type the following commands to set the password back to the default one.

keytool -storepasswd -keystore keystore.jks
Enter keystore password:
New keystore password: changeit
Re-enter new keystore password: changeit
keytool -keypasswd -alias glassfish-instance -keystore keystore.jks
Enter keystore password: changeit
Enter key password for :
New key password for : changeit
Re-enter new key password for : changeit
keytool -keypasswd -alias reporting-instance -keystore keystore.jks
Enter keystore password: changeit
Enter key password for :
New key password for : changeit
Re-enter new key password for : changeit
keytool -keypasswd -alias s1as -keystore keystore.jks
Enter keystore password: changeit
Enter key password for :
New key password for : changeit
Re-enter new key password for : changeit
The most important step, which is missed in the documentation, if you’ve secured this file using anything but ‘changeit‘ then it’ll cause the above asadmin command to fail everytime, even half way through an upgrade.
keytool -storepasswd -keystore cacerts.jks
Enter keystore password:
New keystore password: changeit
Re-enter new keystore password: changeit

Other possible issues which you might encounter:

JRE Path is incorrect

Error: could not open `\AppDynamics\Controller\jre\lib\amd64\jvm.cfg’ The above error is telling you that your JRE path is messed up and it cannot locate the required files.

Solution:

  • check location of above file
  • Edit \AppDynamics\Controller\appserver\glassfish\config\asenv.bat
    • Make sure ‘set AS_JAVA=’ is point to the root of the JRE with the above file.

Error:

Stage [Discover Controller SSL certificate] failed due to
[Task failed: Discovering SSL certificate
on host: HOST
as user: HOST$
with message: Keystore was tampered with, or password was incorrect]

Solution: When upgrade the controller from 4.5 to 4.5, you can set your homemade password in for Glassfish / Keystore / Keys in controller.groovy

keypasswd = "changeit"
storepasswd = "changeit"

but this still failed until we had set all our passwords back to ‘changeit‘ see above on how to do this, then the above controller.groovy file to ‘changeit‘ as well.

Error:

Task failed: Starting Reporting Service
on host: HOST
as user: HOST$
with message: Expected is [RUNNING], but actual result is [STOPPED].

Solution:

Stop the AppDynamics Reporting Service and restart it, then continue the installation back in the AppDynamics 4.5 Enterprise Console.

Good luck out there, upgrading AppDynamics is not an easy task!

We are now looking to move to SaaS because it caused us so many issues.

Windows 10 Update October 2018 / 1809 : No browser website access?

I installed Windows 10 1809 update from here

Bypassing my domain group policy which has greyed out the ‘check for updates’ button. This is slightly annoying in itself.

After installation was completed, I noted that random things were happening when browsing the internet, this was on all the web browsers that I had installed (Chrome, Edge, Firefox). Things like no pages being displayed to browsers working for the first 5 minutes since a fresh reboot, to Chrome crashing and refusing to open again that session. Odd! so I checked whether I could do a

nslookup google.com

From the command prompt.

This was still working as expected. So the network card and settings weren’t the cause of my strife.

I had already upgrade two other machines without any issues, so what was going on? The only difference to the machines were the working ones were laptops using WiFi, the broken one was a desktop using a connected ethernet card.

After a day and a half tweaking / reading the internet, the solution lied with Malwarebytes ! I removed the installation and everything started working again. The installation had recently upgraded itself.

I’ve no idea why and frankly don’t care as it caused me so much hassle to figure out what was going on.

I’ve still got Malwarebytes installed on the two other machines and they are still working as expected, so there must be a buggy version out there that halts the internet access or it detected malware and shut the internet down when using the 1809 update.

Hope this blog post helps someone else too.

Oracle Scheduler Jobs and Email Notifications vs crontab

Question: How do I run scheduled jobs and get a notification via email?

The above is something that our business does all the time, but the answer was, to do that you’ll have to create a crontab direct job on the server.

I thought surely not.. So I thought about it and with a bit of Googling I figured out that the above wasn’t true, it’s possible to do these as scheduled jobs in SQL Developer.

Create Stored Procedure

  • In SQL Developer create a new Procedure
    • Right click on ‘Procedures’ and select ‘New Procedure’, give it a logical name describing what it does, spaces are not allowed, use underscores.
  • Type in your SQL over where the blank Procedure template says ‘NULL;’

Create Scheduled Job

Or jump straight to creating the job if you want to run a block of SQL which doesn’t suit being a Procedure, you can put the SQL directly into the job.

  • Go to the Schema in your database which has the rights to run a scheduled job
  • Under there expand ‘Scheduler’ and ‘Jobs’, right click on ‘Jobs’ and select ‘New Job (Wizard)…’
  • Fill in the field like below
    • Use logical names and give a full description of what the job is doing
    • Either put the SQL in directly in ‘PL/SQL Block’ or select the Procedure you created

JobWizard

When to execute the job?

  • use the ‘When to Execute Job’ drop-down to select ‘Repeating’ and click the pencil this will make the ‘Repeat Interval’ dialog box appear, shown above
  • Select required time and days for the job to run, click ‘OK’.
    • I selected Daily as well as the actual days, just to be sure 🙂
  • Once happy with your choices click ‘Next >’
  • Set Destination to ‘Local’ for it to run on the server, click ‘Next >’
  • Skip over ‘Job Arguments’ to ‘Notification’
  • Now add in ‘job_succeded’ just whilst you are checking the whether your emails are working when the job ran, add it by holding down Ctrl and clicking it, otherwise you will lose your current default options of : job_broken, job_chain_stalled, job_failed, job_over_max_dur, job_sch_lim_reached
  • Move Date: %event_timestamp% from the body to the bottom, as I’ve noticed that it doesn’t create a carriage return after it so will bunch up all the notification output.
    • Also fix the line Error code: by moving the end % back up a line from Error message. This might be a bug on my version of SQL Developer and will be fixed on yours.

From like this:

Retry count: %retry_count%
Error code: %error_code
%Error message: %error_message%

To like this:

Retry count: %retry_count%
Error code: %error_code%
Error message: %error_message%
  • Now Next through the rest of the setting and click ‘Finish’

How to Set up the email side of things

  • Edit and run the following SQL using your System account
BEGIN 

DBMS_SCHEDULER.set_scheduler_attribute('email_server', 'outlook.blah.co.uk:25');
DBMS_SCHEDULER.set_scheduler_attribute('email_sender', 'noreply@blah.co.uk');

END;

That should be it, all that is left to do is to run your job. You can do that by right clicking the job and selecting ‘Run Job…’

Now when people start to automate jobs, they will be visible to your whole team, rather than hidden away on the server in a crontab.

How to Update an expiring / expired Certificate

This is a process which we have to do every 2 years, so I thought I’d better create a post about it, so I don’t lose it.

  • Create CSR (Certificate Signing Request) using local or server IIS – Other methods are avaliable ie: open SSL
    • Open IIS, click ‘Server Certificates’, click ‘Create Certificate Request…’
    • Upload that CSR to your chosen certificate provider and purchase certificate
  • Once request is approved download the Certificate bundle
  • Extract bundle
  • Complete Certificate in IIS where you created the CSR
    • Open IIS, click ‘Server Certificates’, click ‘Complete Certificate Request…’
  • Export Certificate as PFX
    • Open IIS, click ‘Server Certificates’, Right click certificate and Export
  • Import the PFX to the requires servers
    • Run certlm.msc, Personnel, Certificates, Right click and Import, point at PFX
  • Delete the old Expiring Certificate
    • Run certlm.msc, Personnel, Certificates, Find old certificate based on expiry date and delete.
  • Change IIS site binding for 443 or required secure port to point at the new certificate
    • Open IIS, browse in tree to show site, click ‘Bindings’, find SSL port double click and change Certificate in ‘SSL certificate:’ drop down box.

Test new certificate is working

  • Browse to site in Chrome, right click the Secure / Padlock area click ‘Certificate (Valid)’ check Valid From / To entries.

SSL Server’s certificate chain incomplete?

Does your SSLLABS report say ‘This server’s certificate chain is incomplete. Grade capped to B’?

Simple fix:

Concatenate the certificate file with the Intermediate CA.

Open your Certificate file and Intermediate CA in a text editor, copy all of the Intermediate CA file and paste it after the end certificate section.

JOVtnRpn3coVfSR/0rz0XKVXeZGnKztGdIMQhWMTxvZ1UpmRAH2Ab2QnVo1fkPVy
qNSJces5Y/VKpIvLBk5Jj55fvK8ME/9ASa+LtLrIms8iYHl75cupuYZZlg8=
-----END CERTIFICATE----- 

Leaving just the Certificate and the Intermediate Certificate in the file.

Restart your web server and retest in SSLLABS.

SSL Chain issues – Contains anchor

Does your SSLLABS report mention ‘Chain issues – Contains anchor’?

Simple fix:

Remove the Root CA from the concatenated certificate file.

Use a text editor open your Root CA file as well as your Certificate file, check what the Root CA starts and end with and remove that section.

-----BEGIN CERTIFICATE-----
MIIGuDCCBKCgAwIBAgIUUk/B8W400XArhKE/sEK7zHw8kDIwDQYJKoZIhvcNAQEL
BQAwSDELMAkGA1UEBhMCQk0xGTAXBgNVBAoTEFF1b1ZhZGlzIExpbWl0ZWQxHjAc

Blah Blah

JOVtnRpn3coVfSR/0rz0XKVXeZGnKztGdIMQhWMTxvZ1UpmRAH2Ab2QnVo1fkPVy
qNSJces5Y/VKpIvLBk5Jj55fvK8ME/9ASa+LtLrIms8iYHl75cupuYZZlg8=
-----END CERTIFICATE----- 

Leaving just the Certificate and the Intermediate Certificate in the file.

Restart your web server and retest in SSLLABS.

How to copy a Moodle theme

We needed to take a standard Moodle theme ‘Adaptable’, alter it and save that with a new name. This was so it then wasn’t accidentally overwritten with the unaltered standard ‘Adaptable’ theme if we ever had our site rebuilt by our hosting partners. This way we could also package up our extra custom CSS into the theme.

I very much doubt that this post will prove to be useful to you or anyone else.

I’m creating it so I don’t forget how I did it and can reference this in years to come.

Step 1 – Download a fresh copy of your chosen theme

This should give you a .zip file with the theme, unzip it somewhere on your machine.

Step 2 – Find and Replace – Find All

Get hold of a text editor like Notepad ++
In the menu click ‘Search’ / ‘Find in Files…’

Change the following options:

Find what: adaptable
Directory: C:\blah\blah\Adaptable theme 1.8\
Search Mode: Normal
Make sure ‘In all sub-folders‘ is ticked.
Click ‘Find All’

In my case this brings back 1773 hits in 69 files.

Step 3 – Replace in Files

Now you need to do a find and replace to replace the above

Use ‘Find in Files’ again
This time put in the new name of your theme in the ‘Replace with:’ field so lets say ‘adaptable_ray’.
*NOTE never use a minus in the name here xxxx-xxx as it will not work, use an underscore.
Make sure ‘Match case’ is ticked.
Now click ‘Replace in Files’

Step 4 – Theme displayed name

Now you need to Replace the actual displayed name of the theme

In the case of Adaptable this is ‘Adaptable’, so in the ‘Find what:’ field put in ‘Adaptable’ and ‘Replace with:’ ‘Adaptable-Ray’

Step 5 – Tweak JQuery settings

Because of step two you have changed the name of the main themes .js file, but not the actual name of the file. You can either edit /jquery/plugins.php

‘adaptable_ray’ => array(‘files’ => array(‘adaptable_ray.js’)),

To:

‘adaptable_ray’ => array(‘files’ => array(‘adaptable.js’)),

Or Rename the file adaptable.js to adaptable_ray.js

Step 6 – Change the name of the Language file

Rename ‘adaptable_ray\lang\en\theme_adaptable.php’ to ‘theme_adaptable_ray.php’
And any subsequent language packs in \es etc folders.

Step 7 – Change settings files

Rename file ‘adaptable_ray\settings\adaptable_admin_setting_putprops.php’ to ‘adaptable_ray\settings\adaptable_ray_admin_setting_putprops.php’
and
‘adaptable_ray\settings\adaptable_admin_setting_getprops.php’ to ‘adaptable_ray\settings\adaptable_ray_admin_setting_getprops.php’

Step 8 – Change Theme CSS name

Rename the main CSS file from ‘adaptable_ray/style/adaptable.css’ to ‘adaptable_ray/style/adaptable_ray.css’

or

Edit config.php and change the line $THEME->sheets = array( ‘adaptable_ray’,
to $THEME->sheets = array( ‘adaptable’,

Step 9 – Change the folder name

Now change the folder name to your new name in lowercase the same name as in Step 3.

Step 10 – Zip up the theme

Zip up the theme as ‘adaptable_ray.zip’

Step 11 – Deploy the theme

Deploy the theme to your local installation and check it all works.